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To our families 


Preface 


This book has been designed for a final year undergraduate course in stochastic 
processes. It will also be suitable for mathematics undergraduates and others 
with interest in probability and stochastic processes, who wish to study on their 
own. 

The main prerequisite is probability theory: probability measures, random 
variables, expectation, independence, conditional probability, and the laws of 
large numbers. The only other prerequisite is calculus. This covers limits, series, 
the notion of continuity, differentiation and the Riemann integral. Familiarity 
with the Lebesgue integral would be a bonus. A certain level of fundamental 
mathematical experience, such as elementary set theory, is assumed implicitly. 

Throughout the book the exposition is interlaced with numerous exercises, 
which form an integral part of the course. Complete solutions are provided at 
the end of each chapter. Also, each exercise is accompanied by a hint to guide 
the reader in an informal manner. This feature will be particularly useful for 
self-study and may be of help in tutorials. It also presents a challenge for the 
lecturer to involve the students as active participants in the course. 

A brief introduction to probability is presented in the first chapter. This is 
mainly to fix terminology and notation, and to provide a survey of the results 
which will be required later on. However, conditional expectation is treated in 
detail in the second chapter, including exercises designed to develop the nec- 
essary skills and intuition. The reader is strongly encouraged to work through 
them prior to embarking on the rest of this course. This is because conditional 
expectation is a key tool for stochastic processes, which often presents some 
difficulty to the beginner. 

Chapter 3 is about martingales in discrete time. We study the basic prop- 
erties, poe also some more advanced ones a LOR PMNS times and the Optional 
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Doob’s inequalities and convergence results. Chapter 5 is devoted to time- 
homogenous Markov chains with emphasis on their ergodic properties. Some 
important results are presented without proof, but with a lot of applications. 
However, Markov chains with a finite state space are treated in full detail. 
Chapter 6 deals with stochastic processes in continuous time. Much emphasis 
is put on two important examples, the Poisson and Wiener processes. Various 
properties of these are presented, including the behaviour of sample paths and 
the Doob maximal inequality. The last chapter is devoted to the It6 stochastic 
integral. This is carefully introduced and explained. We prove a stochastic ver- 
sion of the chain rule known as the Ito formula, and conclude with examples 
and the theory of stochastic differential equations. 

It is a pleasure to thank Andrew Carroll for his careful reading of the final 
draft of this book. His many comments and suggestions have been invaluable 
to us. We are also indebted to our students who took the Stochastic Analysis 
course at the University of Hull. Their feedback was instrumental in our choice 
of the topics covered and in adjusting the level of exercises to make them 
challenging yet accessible enough to final year undergraduates. 

As this book is going into its 3rd printing, we would like to thank our 
students and readers for their support and feedback. In particular, we wish 
to express our gratitude to Iaonnis Emmanouil of the University of Athens 
and to Brett T. Reynolds and Chris N. Reynolds of the University of Wales 
in Swansea for their extensive and meticulous lists of remarks and valuable 
suggestions, which helped us to improve the current version of Basic Stochastic 
Processes. 

We would greatly appreciate further feedback from our readers, who are 
invited to visit the Web Page http://www. hull.ac.uk/php/mastz/bsp. html 
for more information and to check the latest corrections in the book. 


Zdzistaw Brzezniak and Tomasz Zastawniak 
Kingston upon Hull, June 2000 
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Review of Probability 


In this chapter we shall recall some basic notions and facts from probability 
theory. Here is a short list of what needs to be reviewed: 


1) Probability spaces, o-fields and measures; 


) 

2) Random variables and their distributions; 
3) Expectation and variance; 

4) The o-field generated by a random variable; 
5) Independence, conditional probability. 


The reader is advised to consult a book on probability for more information. 


1.1 Events and Probability 


Definition 1.1 


Let 2 be a non-empty set. A o-field F¥ on 2 is a family of subsets of $2 such 
that 


1) the empty set @ belongs to F; 
2) if A belongs to F, then so does the complement 2 \ A; 
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3) if Ay, Ag,... is a sequence of sets in F, then their union A; U Aj U--: also 
belongs to F. 


Example 1.1 


Throughout this course R will denote the set of real numbers. The family of 
Borel sets F = B(R) is a o-field on R. We recall that B(R) is the smallest 
g-field containing all intervals in R. 


Definition 1.2 
Let F be a o-field on $92. A probability measure P is a function 


P:F > (0,1) 
such that 
1) P(2) =1; 


2) if A,,Ao,... are pairwise disjoint sets (that is, A;M A; = 0 for i # 7) 
belonging to F, then 


PAR UA UH) SPO) Paae: . 


The triple (2,7,P) is called a probability space. The sets belonging to F 
are called events. An event A is said to occur almost surely (a.s.) whenever 


P(A) =1. 


Example 1.2 


We take the unit interval 2 = [0,1] with the o-field F = B((0,1]) of Borel 
sets B C [0,1], and Lebesgue measure P = Leb on [0,1]. Then (00,F,P) is a 
probability space. Recall that Leb is the unique measure defined on Borel sets 


such that 
Lebja, b] = b-a 


for any interval [a,b]. (In fact Leb can be extended to a larger o-field, but we 


shall need Borel sets only.) 


Exercise 1.1 


Show that if A,, Ao,... is an expanding sequence of events, that is, 


Ai CAngC:::, 
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then 
P(A; U Ao U---) = iin P{Ay,): 
n— Oo 
Similarly, if A,, Ag,... is a contracting sequence of events, that is, 
A, D Ao TEE, 5 
then 
P(A; NA2N---) = lim P(A,). 
n— Ooo 
Hint Write Ai U Az U--- as the union of a sequence of disjoint events: start with 


Ai, then add a disjoint set to obtain A; U Ag, then add a disjoint set again to obtain 
A; U A2 U Az, and so on. Now that you have a sequence of disjoint sets, you can use 
the definition of a probability measure. To deal with the product AiMA2M--- write 
it aS a union of some events with the aid of De Morgan’s law. 


Lemma 1.1 (Borel—Cantelli) 
Let A;,A2,... be a sequence of events such that P(A,) + P(A2)+--: < co 
and let B, = An U Anyi U-::. Then P(ByN B2N-:--) =0. 


Exercise 1.2 


Prove the Borel—Cantelli lemma above. 


Hint B,,Bo2,... is a contracting sequence of events. 


1.2 Random Variables 


Definition 1.3 
If F is ao-field on $2, then a function € : f{2 > R is said to be F-measurable if 
{EE BS eF 


for every Borel set B € B(R). If (92,7, P) is a probability space, then such a 
function € is called a random variable. 


Remark 1.1 


A short-hand notation for events such as {€ € B} will be used to avoid clutter. 
To be precise, we should write 


{wE:&(w) Ee BS 
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in place of {€ € B}. Incidentally, {€ € B} is just a convenient way of writing 
the inverse image €~! (B) of a set. 


Definition 1.4 


The o-field o (€) generated by a random variable € : 2 > R consists of all sets 
of the form {€ € B}, where B is a Borel set in R. 


Definition 1.5 


The o-field o {€; : i € I} generated by a family {€; : i € I} of random variables 
is defined to be the smallest o-field containing all events of the form {&; € B}, 
where B is a Borel set in R andi € I. 


Exercise 1.3 


We call f : R > Ra Borel function if the inverse image f~!(B) of any Borel 
set B in R is a Borel set. Show that if f is a Borel function and € is a random 
variable, then the composition f (€) is o (€)-measurable. 


Hint Consider the event {f (€) € B}, where B is an arbitrary Borel set. Can this 
event be written as {€ € A} for some Borel set A? 


Lemma 1.2 (Doob—Dynkin) 


Let € be a random variable. Then each o (€)-measurable random variable 7 can 
be written as 


n= f (&) 


for some Borel function f : R—- R. 


The proof of this highly non-trivial result will be omitted. 


Definition 1.6 
Every random variable € : (2 + R gives rise to a probability measure 
P;(B) =P {Ee B} 


on R defined on the o-field of Borel sets B € B(IR). We call Pe the distribution 
of €. The function Fe : R — [0, 1] defined by 


Fe (zt) = P{E< 2} 


~a ew 
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Exercise 1.4 


Show that the distribution function F¢ is non-decreasing, right-continuous, and 


_Jim Fe (x) = 0, _ lim. Fe (x) = 1. 


Hint For example, to verify right-continuity show that F¢(rn) — F¢(x) for any de- 
creasing sequence z, such that zr, — zx. You may find the results of Exercise 1.1 
useful. 


Definition 1.7 
If there is a Borel function f¢ : R — R such that for any Borel set B C R 
P{ée B} =| f(a) dz, 


then € is said to be a random variable with absolutely continuous distribution 
and f¢ is called the density of €. If there is a (finite or infinite) sequence of 


pairwise distinct real numbers 21, 22,... such that for any Borel set B C R 
Pie By=) Ple=a}, 
zi€B 
then € is said to have discrete distribution with values z1,22,... and mass 


P{€=2;} at xj. 


Exercise 1.5 
Suppose that € has continuous distribution with density fe. Show that 


ZF (2) = fe 


if fe is continuous at z. 


Hint Express F¢ (x) as an integral of fe. 


Exercise 1.6 


Show that if € has discrete distribution with values z1,72,... , then F¢ is 
constant on each interval (s,¢] not containing any of the z;’s and has jumps of 
size P {€ = 2;} at each 2;. 


Hint The increment F¢ (t) — F¢ (s) is equal to the total mass of the z;’s that belong 
to the interval [s, ¢). 
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Definition 1.8 


The joint distribution of several random variables €),...,€, is a probability 
measure P¢,,...¢, on IR” such that 


Pey,..& (B) = P{(Eiy---s&n) € B} 


gece 


chat 
P{(Gys6s) € B}= | Fee sintee (Pigseag@y) dz, -:-d&n 
B 


or any Borel set B in R”, then fe,,¢, is called the joint density of ,...,&n. 


nr 


Definition 1.9 
4 random variable € : {2 > R is said to be integrable if 
iy idP Ss, 
Q 
Chen 
B(@) = | ¢ aP 
Q 


‘xists and is called the ezpectation of €. The family of integrable random vari- 
tbles € : 22 — R will be denoted by L! or, in case of possible ambiguity, by 
CQ ck): 


=xample 1.3 


Che indicator function 1,4 of a set A is equal to 1 on A and 0 on the complement 
2\A of A. For any event A 


E (1a) = (gP= PC): 
Q 
Ne say that 7: 2 > Ris a step function if 
Le aunere 
i=l 


vhere 71,...,%n are real numbers and Aj,...,A, are pairwise disjoint events. 


Chen S - 
E(n) = [ nar = Son fla dP = So mP (Ai). 
i=l i=l 
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Exercise 1.7 


Show that for any Borel function h: R > R such that h(€) is integrable 
B(h(6)) = | h(a) dP (@). 
R 


Hint First verify the equality for step functions h : R > R, then for non-negative ones 
by approximating them by step functions, and finally for arbitrary Borel functions by 
splitting them into positive and negative parts. 


In particular, Exercise 1.7 implies that if € has an absolutely continuous 
distribution with density fe, then 


+00 


E(h(€)) = i h(a) fe (x) de. 


— OO 


If € has a discrete distribution with (finitely or infinitely many) pairwise distinct 
values 71, 22,..., then 


E(h(€)) = Doh (as) P{E = zit. 


Definition 1.10 


A random variable € : {2 > R is called square integrable if 
| I€|? dP < co. 
2 
Then the variance of € can be defined by 
var(é) = [ (€- BO)? aP, 
Q 
The family of square integrable random variables € : {2 > R will be denoted 


by L?(92,F,P) or, if no ambiguity is possible, simply by L?. 


Remark 1.2 
The result in Exercise 1.8 below shows that we may write /(€) in the definition 


of variance. 


Exercise 1.8 


Show that if € is a square integrable random variable, then it is integrable. 
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Hint Use the Schwarz inequality 
[E (En)|’ < E (€’) E (n’) 


with an appropriately chosen 7. 


Exercise 1.9 


(1.1) 


Show that if 7 : 2 — [0,00) is a non-negative square integrable random vari- 


able, then = 
E(n’) = 2 | tP(n > t) dt. 
0 


Hint Express E(n”) in terms of the distribution function F,,(t) of 7 and then integrate 


by parts. 


1.3 Conditional Probability and Independence 


Definition 1.11 


For any events A, B € F such that P(B) 4 0 the conditional probability of A 


given B is defined by 
P(ANB) 


Exercise 1.10 


Prove the total probability formula 
P(A) = P(A|B1) P(B1) + P(A|B2)P(B2) ++ 


for any event A € F and any sequence of pairwise disjoint events B,, Bo,... 


such that B; U Bp U--- = 2 and P(B,) # 0 for any n. 


Hint A=(ANB,)U(AN Bo) U--: ; 


Definition 1.12 
Two events A,B € F are called independent if 
P(ANB) = P(A)P(B). 
In general, we say that n events A,,...,An € F are independent if 


P(Aj, N Ai, Ase OAg,) = P(A;,)P(Ai,)--- P(Ai,) 


EF 
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for any indices 1 <7) < ig < +++ << ap <n. 


Exercise 1.11 
Let P(B) # 0. Show that A and B are independent events if and only if 
P(A|B) = P(A). 


Hint If P(B) # 0, then you can divide by it. 


Definition 1.13 


Two random variables € and 7 are called independent if for any Borel sets 
A, B € B(R) the two events 


{EEA} and {ne B} 


are independent. We say that n random variables £1,...,&, are independent if 
for any Borel sets B,,...,Bn € B(R) the events 


{& € By},...,{&, € Br} 


are independent. In general, a (finite or infinite) family of random variables 
is said to be independent if any finite number of random variables from this 
family are independent. 


Proposition 1.1 


If two integrable random variables €,7 : (2 + R are independent, then they are 
uncorrelated, i.e. 


E(&n) = E(€)E(n), 


provided that the product £7 is also integrable. If €,,...,& : 2 —74 R are 
independent integrable random variables, then 


B(f:€2-+-&n) = E (1) B (f2)---E (én); 


provided that the product &)&2---€, is also integrable. 


Definition 1.14 


Two o-fields G and H contained in F are called independent if any two events 


A€G and BEH 
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are independent. Similarly, any finite number of o-fields G,,...,G, contained 
in F are independent if any n events 


Aj COG iitedg Aare Ge 


are independent. In general, a (finite or infinite) family of o-fields is said to be 
independent if any finite number of them are independent. 


Exercise 1.12 


Show that two random variables € and 7 are independent if and only if the 
o-fields o (€) and a (7) generated by them are independent. 


Hint The events in o (€) and o (7) are of the form {€ € A}, and {7 € B}, where A 
and B are Borel sets. 


Sometimes it is convenient to talk of independence for a combination of 
random variables and o-fields. 


Definition 1.15 
We say that a random variable € is independent of a o-field G if the o-fields 
o(€) and G 


are independent. This can be extended to any (finite or infinite) family con- 
sisting of random variables or g-fields or a combination of them both. Namely, 
such a family is called independent if for any finite number of random variables 
€,,...,&m and o-fields Gj,...,Gn from this family the o-fields 


O (Ei) 9x0 290 (Em) Giese; Gn 


are independent. 


1.4 Solutions 


Solution 1.1 
If A; C Ao C-:-, then 
A, U Ap U--++ = Ay U (Ag \ At) U (A3 \ An) Uses, 
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where the sets A), Az \ Aj, A3 \ Ag,... are pairwise disjoint. Therefore, by the 
definition of probability measure 


P(A, U Ag U---) = P(A, U(A2 \ A1) U(A3 \ Aa) Ue: ) 
Ap ere (Ap a) ee (Ag Ag aes 
= lim P(A,). 


nN— CoO 


The last equality holds because the partial sums in the series above are 
P(A) + P (Ap \ Ay) +--+ + P(An \ An-1) = P(A, U---UAn) 
= PCAs) 
If Ay D Ao D-::: , then the equality 
P(A, NAgN:-:) = lim P (An) 
m— Oo 
follows by taking the complements of A, and applying De Morgan’s law 
Q\ (Ar NM Ag N-+ +) =(A\ A) U(R\ A2)U-- . 


Solution 1.2 


Since B, is a contracting sequence of events, the results of Exercise 1.1 imply 
that 


P(B,NB2n-::) = lim P (Bn) 
= lim P (An U An+1 Us) 
lim (P (An) + P (Ani) +++") 


0. 


IA 


The last equality holds because the series )>>, P(An) is convergent. The 
inequality above holds by the subadditivity property 


P (An U Angi U'::) < P(An) + P (Angi) +:°° - 
It follows that P(B, 1 B2N---) =0. 


Solution 1.3 


If B is a Borel set in R and f : R > Ris a Borel function, then f—! (B) is also 
a Borel set. Therefore 


{f (€) € B} = {€€ f-* (B)} 
belongs to the o-field o (€) generated by €. It follows that the composition f (€) 


ie 7 (£\-meacnrahle 
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Solution 1.4 
Ifx <y, then {€ <2} Cc {€<y}, so 
Fy (tz) = P{E<a}< P{E<y} = (y). 


This means that F¢ is non-decreasing. 
Next, we take any sequence 2; > r2 > --- and put 


lim 2, = 72. 
n—-+oo 


Then the events 
{CS 1} DAG Ss to} De 


form a contracting sequence with intersection 
{E<z}={E<Sai}N{E<a2}n---. 
It follows by Exercise 1.1 that 
Fe (x) =P{E<z}= jim P {é < In} = Jim Fe (Tp). 


This proves that F¢ is right-continuous. 


Since the events 
(oS Sl} DAG es =o) Oe 


form a contracting sequence with intersection @ and 
Le STC 16 52) Cee 
form an expanding sequence with union 2, it follows by Exercise 1.1 that 


lim Fy (zr) = dim Fe (—n) = jim P {é < —n} = P(6) = 0, 


lim Fe (x) = lim Fe (n) = lim PLE <n} =P (2) =, 


since F¢ is non-decreasing. 


Solution 1.5 


If € has a density f¢, then the distribution function F¢ can be written as 
0) 
Fe(z)=P{E<a}= [  fe(v) dy. 
—0o 
Therefore, if fe is continuous at z, then F¢ is differentiable at x and 


ay (x) = fe (x) . 
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Solution 1.6 


If s < ¢ are real numbers such that z; ¢ (s,t] for any 2, then 
Fe(t) — Fe(s) = P{€é <t} — P{E < 8} = P{E € (s, t]} = 0, 


i.e. Fg(s) = Fe(t). Because Fy is non-decreasing, this means that F¢ is constant 
on (s,t]. To show that F; has a jump of size P{€ = z;} at each z;, we compute 


pe (t) — jim Fe(s) = im P{E<t} - = P{E< s} 
P{E< ai} — P{E< 2} = P{E = aj}. 


Solution 1.7 


If h is a step function, 
n 
h= > hila,, 
i=1 


where hy,...,A, are real numbers and A;,...,A, are pairwise disjoint Borel 
sets covering R, then 


E(h(é)) = So AB (la; (©) = Do P(E € Ai} 
a=1 i=l 


_ Shire d= fre dP. (2) = [ rte) iP 


Next, any non-negative Borel function h can be approximated by a non- 
decreasing sequence of step functions. For such an A the result follows by the 
monotone convergence of integrals. Finally, this implies the desired equality 
for all Borel functions h, since each can be split into its positive and negative 
parts, h = ht —h-, where ht,h~ > 0. 


Solution 1.8 

By the Schwarz inequality (1.1) with n = 1, if € is square integrable, then 
[E (lel = (Aen < B(1’) E(@) = £(€) < om, 

i.e. € is integrable. 


Solution 1.9 
Let F(t) = P {n < t} be the distribution function of 7. Then 


E(n?) = / 7 t?dF(t). 


0 
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Since P(n > t) = 1 — F(t), we need to show that 


- t? dF(t) = 2 [+a — F(t)) dt (1.2) 
0 0 


First, let us establish a version of (1.2) with co replaced by a finite number a. 
Integrating by parts, we obtain 


[ ear = [ Pare =i) 

0 0 

5-2 / t(F(t) — 1) dt 
0 


II 
oH 
iw) 
—™~ 
ty 
“—™~ 
o 
wa 
| 
a 
wa 


—a?(1 — F(a)) +2 [ t(1 — F(t))dt. (1.3) 


0 
We see that (1.2) follows from (1.3), provided that 


a’(1-F(a)) 30, asa-+o. (1.4) 
But 
0 <a*(1— F(a)) =a°P(n > a) < (n+1)?P(n > n) < 4n?P(n > 2), 


where n is the integer part of a, and 


E(7*) =F i n’dP < OO. 
ko ¥ (kSnsk+1} 
Hence, 
mP(nzn) < | raP= > | 7’ dP > 0 (1.5) 
{n2n} kan 1 (k<n<kt+1} 


as m —> oo, which proves (1.4). 


Solution 1.10 
Since B} UBgU::-=N2, 
A= AN(B,UB2U---)=(ANB)U(AN Ba) U--- , 


where 

(AN Bi) N(ANB;) = AN(BiNB;) =ANO=8. 
By countable additivity 

P(A) = P((AN B,)U(AN By) U-::) 
P(ANB,)+P(AN Bo) +::: 
P(A|B,) P(Bi) + P(A| Bo) P(B2) +--+: . 


II 
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Solution 1.11 
If P(B) #0, then A and B are independent if and only if 


P(ANB 
P= Say 
In turn, this equality holds if and only if P(A) = P(A|B). 
Solution 1.12 
The o-fields o (€) and o (7) consist, respectively, of events of the form 


{EEA} and {ne B}, 


where A and B are Borel sets in R. Therefore, o (€) and o (7) are independent 
if and only if the events {€ € A}, and {n € B} are independent for any Borel 
sets A and B, which in turn is equivalent to € and n being independent. 


2 


Conditional Expectation 


Conditional expectation is a crucial tool in the study of stochastic processes. 
It is therefore important to develop the necessary intuition behind this notion, 
the definition of which may appear somewhat abstract at first. This chapter is 
designed to help the beginner by leading him or her step by step through several 
special cases, which become increasingly involved, culminating at the general 
definition of conditional expectation. Many varied examples and exercises are 
provided to aid the reader’s understanding. 


2.1 Conditioning on an Event 


The first and simplest case to consider is that of the conditional expectation 
E (€|B) of a random variable € given an event B. 


Definition 2.1 


For any integrable random variable € and any event B € F such that P(B) 40 
the conditional expectation of € given B is defined by 


E(é|B) = ra i: é dP. 
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Example 2.1 


Three coins, 10p, 20p and 5O0p are tossed. The values of those coins that land 
heads up are added to work out the total amount €. What is the expected total 
amount € given that two coins have landed heads up? 

Let B denote the event that two coins have landed heads up. We want to 
find EF’ (€|B). Clearly, B consists of three elements, 


B = {HHT, HTH, THH}, 


each having the same probability e (Here H stands for heads and T for tails.) 
The corresponding values of € are 


€(HHT) = 10+ 20 = 30, 
€(HTH) = 10+50 = 60, 
¢(THH) = 20 +50 = 70. 


Therefore 
1 1/30 60 = 70 
Ean) = iPS ee ee | Se 53, 


Exercise 2.1 


Show that E (£|2) = E(€). 


Hint The definition of F (€) involves an integral and so does the definition of F (€|{2). 
How are these integrals related? 
Exercise 2.2 


Show that if 
1 forweA 


0 forw¢ A 
(the zndicator function of A), then 
E(14|B) = P(A|B), 
where 
P(ANB) 
P(B) 
is the conditional probability of A given B. 


P(A|B) = 


Hint Write [14 dP as P(AN B). 
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2.2 Conditioning on a Discrete Random 
Variable 


The next step towards the general definition of conditional expectation involves 
conditioning by a discrete random variable n with possible values yj, y2,... such 
that P{n = yn} 4 0 for each n. Finding out the value of 7 amounts to finding 
out which of the events {7 = yn} has occurred or not. Conditioning by 7 should 
therefore be the same as conditioning by the events {7 = y,}. Because we do 
not know in advance which of these events will occur, we need to consider all 
possibilities, involving a sequence of conditional expectations 


E(€\{in=y}), 2 (El {7 = y2}),... . 


A convenient way of doing this is to construct a new discrete random variable 
constant and equal to E (€| {7 = yn}) on each of the sets {7 = yn}. This leads 
us to the next definition. 


Definition 2.2 


Let € be an integrable random variable and let 7 be a discrete random variable 
as above. Then the conditional expectation of € given 7 is defined to be a random 
variable E(€|n) such that 


E(E|n)(w) = E(El {7 = yn}) if nw) = yn 


for any n=1,2,.... 


Example 2.2 


Three coins, 10p, 20p and 50p are tossed as in Example 2.1. What is the 
conditional expectation E (€|n) of the total amount € shown by the three coins 
given the total amount 7 shown by the 10p and 20p coins only? 

Clearly, 7 is a discrete random variable with four possible values: 0, 10, 20 
and 30. We find the four corresponding conditional expectations in a similar 
way as in Example 2.1: 


E(E\{n=0})=25, E(€| {n = 10}) = 35, 
E (in = 20})=45, E(€|{n = 30}) = 55. 


Therefore 
25 if n(w) = 0, 
_ J 35 if n(w) = 10, 
E (€|n) (w) = 45 if n(w) = 20. 
55 if n(w) = 30. 
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Example 2.3 


Take §2 = [0,1] with the o-field of Borel sets and P the Lebesgue measure on 
(0, 1]. We shall find E(€|n) for 


1 
(x) = 22°, n(x) = 2 if e (3,4), 
0 


Clearly, 7 is discrete with three possible values 1, 2,0. The corresponding events 
are 


{n = 1} —s (0, ae 
{n = 2} = (4, 3), 
{n = 0} = (3,]] 


For x € (0, §] 
1 1 fs 2 2 
B(EIn)(2) = B(E\(0,$)) = ¢ f° 2xds = = 
3 40 
For x € (4, 2) 
1 2 1 2 1 
E(€ln)(x) = E(El(z,3)) = 7 | 2a°dr = =. 
3/4 a 
And for z € (2, 1] 
; Loe 38 
B(éln)(z) = E31) = 7 [, 22*de = vk 
3.°3 
The graph of E(£|7) is shown in Figure 2.1 together with those of € and 7. 


Exercise 2.3 


Show that if 7 is a constant function, then E(€|n) is constant and equal to 


E(é). 


Hint The event {7 =c} must be @ or 2 for any cE R. 


Exercise 2.4 


Show that 
P(A|B) ifweB 


E(1a4|1B)(w) = { P(A|Q \ B) ifw¢é B 


for any B such that 1 4 P(B) £ 0. 
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Figure 2.1. The graph of EF (€|7) in Example 2.3 


Hint How many different values does 1g take? What are the sets on which these 
values are taken? 


Exercise 2.5 


Assuming that 7 is a discrete random variable, show that 


E(E(€|n)) = E(é). 


| eemar = | ear 
B B 


for any event B on which 7 is constant. The desired equality can be obtained by 
covering $2 by countably many disjoint events of this kind. 


Hint Observe that 


Proposition 2.1 


If € is an integrable random variable and 77 is a discrete random variable, then 


1) E(€|n) is o (7)-measurable; 


2) For any A € a(n) 
i: E(éln) dP = / € dP (2.1) 
A A 


Proof 


Suppose that 7 has pairwise distinct values yi, y2,.... Then the events 


{n=1},{n = ye},... 
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are pairwise disjoint and cover (2. The o-field o (7) is generated by these events, 
in fact every A € o (7) is acountable union of sets of the form {7 = y,}. Because 
E (€|n) is constant on each of these sets, it must be o (7)-measurable. 

For each n we have 


| BnaP 
{n=yn} 


/ E(é| {n = yn}) dP 
{n=yn} 


/ E dP. 
{n=yn} 


Since each A € o(7) is a countable union of sets of the form {7 = yn}, which 
are pairwise disjoint, it follows that 


/ B(gin) aP = ff ear 


as required. U1 


2.3 Conditioning on an Arbitrary Random 
Variable 


-roperties 1) and 2) in Proposition 2.1 provide the key to the definition of the 
‘onditional expectation given an arbitrary random variable 7. 


Jefinition 2.3 


set € be an integrable random variable and let 7 be an arbitrary random 
ariable. Then the conditional expectation of € given 7 is defined to be a random 
ariable E(€|7) such that 


1) E(E|n) is o (n)-measurable; 
2) For any A € a(n) 


| Belmar = | car 


‘emark 2.1 


/e can also define the conditional probability of an event A € F given 7 by 


here 1,4 is the indicator function of A. 
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Do the conditions of Definition 2.3 characterize E(€|n) uniquely? The 
lemma below implies that E (€|n) is defined to within equality on a set of 
full measure. Namely, 


if€=é¢' as. then E(€ln) = E(E'|n) as. (2.2) 


The existence of E (€|7) will be discussed later in this chapter. 


Lemma 2.1 


Let (2,7, P) be a probability space and let Y be a o-field contained in F. If € 
is a G-measurable random variable and for any B € G 


[ ear =0, 


then € = 0 a.s. 


Proof 
Observe that P {€ ><} =0 for any e > 0 because 


0<eP{E>e}= | cap < | €dP =0. 
{€>¢} {€>¢} 


The last equality holds, since {€ > ¢€} € G. Similarly, P{€ < —e} = 0 for any 
€ > 0. As a consequence, 


P{-e<E<e}=1 


for any € > 0. 
Let us put 
Ag ={-1 <6 < 2}. 


Then P(A,) = 1 and 
{€ =0} = {} An. 


n=1 


Because the A, form a contracting sequence of events, it follows that 
P{€=0}= lim P(A,) = 1, 
m—>OO 


as required. UO 
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One difficulty involved in Definition 2.3 is that no explicit formula for & (€|7) 
is given. If such a formula is known, then it is usually fairly easy to verify 
conditions 1) and 2). But how do you find it in the first place? The examples 
and exercises below are designed to show how to tackle this problem in concrete 


Cases. 


Example 2.4 


Take 2 = [0,1] with the o-field of Borel sets and P the Lebesgue measure on 
(0, 1]. We shall find E(€|7) for 


2 if « € (0,5), 
x if xe (5,1). 


Oe ee Ce { 


Here 77 is no longer discrete and the general Definition 2.3 should be used. 
First we need to describe the o-field o(7). For any Borel set B C [3,1] we 


have 
B= {n€ B} € a(n) 


and 
(0,5) UB = {n€ B}U {n= 2} € a(n). 
In fact sets of these two kinds exhaust all elements of o (7). The inverse image 
{7 € C} of any Borel set C' C R is of the first kind if 2 ¢ C and of the second 
kind if2€C. 
If E(€|n) is to be o (n)-measurable, it must be constant on (0, 5) because 7 
is. If for any z € (0, 5) 


E(Eln)(x) = E(é|[0, 5)) 
1 
7 P((0, 5)) a 7 


7 
2 
i 2x7 dx 
0 


“ 


| = bye] 


then 


/ B(Eln)(2) de = / E(x) de, 
(0,5) (0,4) 


i.e. condition 2) of Definition 2.3 will be satisfied for A = [0, 5). 
Moreover, if E(€|7) = € on [5,1], then of course 


| B(€n)(a)ax = | &(2) ae 
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for any Borel set B C [$, 1). 
Therefore, we have found that 


1 if 0, 4), 
E(E|n)(x) = se — 4 


Because every element of a(n) is of the form B or (0, 5) U B, where B C [5 1] 
is a Borel set, it follows immediately that conditions 1) and 2) of Definition 2.3 
are satisfied. The graph of E(€|n) is presented in Figure 2.2 along with those 
of € and 7. 


Figure 2.2. The graph of F (€|7) in Example 2.4 


Exercise 2.6 


Let 2 = [0,1] with Lebesgue measure as in Example 2.4. Find the conditional 
expectation FE (€|n) if 


E(x) = 22”, n(x) = 1—|2r —- 1]. 


Hint First describe the o-field generated by 7. Observe that 7 is symmetric about >. 
What does it tell you about the sets in o (7)? What does it tell you about F (|) if it 
is to be o (7)-measurable? Does it need to be symmetric as well? For any A in a (7) 


try to transform iP €dP to make the integrand symmetric. 


Exercise 2.7 


Let 2 be the unit square [0,1] x [0,1] with the o-field of Borel sets and P the 
Lebesgue measure on [0,1] x [0,1]. Suppose that € and 7 are random variables 
on §2 with joint density 

fen(z,y) =@2+y 
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for any z,y € [0,1], and fe)(z,y) = 0 otherwise. Show that 


_ 2+3n 
— 346 


E (€\n) 


Hint It suffices (why?) to show that for any Borel set B 


/ cap = | aaa AP 
{ne B} {nEB} 707] 


Try to express each side of this equality as an integral over the square (0, 1] x (0, 1] 
using the joint density fe,,(2, y). 


Exercise 2.8 


Let §2 be the unit square (0, 1] x [0, 1] with Lebesgue measure as in Exercise 2.7. 
Find EF (€|n) if € and 7 are random variables on 92 with joint density 


3 
fem(z,y) = 5 (oe ae y*) 
for any z,y € [0,1], and f¢,,(z, y) = 0 otherwise. 


Hint This is slightly harder than Exercise 2.7 because here we have to derive a formula 
for the conditional expectation. Study the solution to Exercise 2.7 to find a way of 
obtaining such a formula. 


Exercise 2.9 


Let 92 be the unit disc {(z,y) : 2? + y? < 1} with the o-field of Borel sets and 
P the Lebesgue measure on the disc normalized so that P (2) = 1, ie. 


para2 ff aca 


for any Borel set A C §2. Suppose that € and 7 are the projections onto the x 
and y axes, 
E(x,y)=2, n(z,y)=y 


for any (x,y) € 92. Find E (€?|n). 


Hint What is the joint density of € and 7? Use this density to transform the integral 


/ €7 dP 
{ne€B} 


for an arbitrary Borel set B so that the integrand becomes a function of 7. How is 
this function of 7 related to E (€7|n)? 
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2.4 Conditioning on a o-Field 


We are now in a position to make the final step towards the general definition 
of conditional expectation. It is based on the observation that FE (€|n) depends 
only on the o-field o (7n) generated by 7, rather than on the actual values of 7. 


Proposition 2.2 
If a(n) = a(n’), then E(E|n) = E(€|n’) a.s. (Compare this with (2.2).) 


Proof 
This is an immediate consequence of Lemma 2.1. 0 
Because of Proposition 2.2 it is reasonable to talk of conditional expectation 


given a o-field. The definition below differs from Definition 2.3 only by using an 
arbitrary o-field G in place of a o-field o (7) generated by a random variable 7. 


Definition 2.4 


Let € be an integrable random variable on a probability space (92,7, P), and 
let G be a o-field contained in F. Then the conditional expectation of € given 
G is defined to be a random variable F (€|G) such that 


1) E(€|G) is G-measurable; 
2) For any AEG 
i E(€\G) dP = / €dP. (2.3) 
A A 


Remark 2.2 


The conditional probability of an event A € F given a o-field G can be defined 
by 
P(A|G) = E (1a4|Q), 


where 1, is the indicator function of A. 


The notion of conditional expectation with respect to a o-field extends 
conditioning on a random variable 7 in the sense that 


E (€|o(n)) = E (|), 


where a(7) is the o-field generated by 7. 
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Proposition 2.3 


E(€|G) exists and is unique in the sense that if € = €' a.s., then E(€|G) = 
E(€'|G) as. 


Proof 


Existence and uniqueness follow, respectively, from Theorem 2.1 below and 
Lemma 2.1. 0 


Theorem 2.1 (Radon—Nikodym) 


Let (2,7, P) be a probability space and let G be a o-field contained in F. 
Then for any random variable € there exists a G-measurable random variable 


¢ such that 
[sap = f ca 


The Radon—Nikodym theorem is important from a theoretical point of view. 
However, in practice there are usually other ways of establishing the existence 
of conditional expectation, for example, by finding an explicit formula, as in 
the examples and exercises in the previous section. The proof of the Radon-— 
Nikodym theorem is beyond the scope of this course and is omitted. 


for each A € G. 


Exercise 2.10 
Show that if G = {0,Q}, then E(E|G) = E(€) as. 


Hint What random variables are G-measurable if G = {@, 2}? 

Exercise 2.11 

Show that if € is G-measurable, then E(€|G) = € a.s. 

Hint The conditions of Definition 2.4 are trivially satisfied by € if € is G-measurable. 
Exercise 2.12 


Show that if B € G, then 
E(E (é|G) |B) = E(¢|B). 


Hint The conditional expectation on either side of the equality involves an integral 
aver RB How are these integrals related to one another? 
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2.5 General Properties 


Proposition 2.4 
Conditional expectation has the following properties: 
1) E(ag + b6|G) = aE (é|G) + bE(¢|G) (linearity); 
2) E(E(E|G)) = E(é); 
E(E¢\G) = €E(C|G) if € is G-measurable (taking out what is known); 


3) 
4) E(€|G) = E(€) if € is independent of G (an independent condition drops 
out); 


5) E(E(E|G)|H) = E(€|H) if H C G (tower property); 
6) If € > 0, then E(E|G) > O (positivity). 


Here a,b are arbitrary real numbers, €,¢ are integrable random variables on a 
probability space (2,7, P) and G, H are o-fields on §2 contained in ¥. In 3) we 
also assume that the product €¢ is integrable. All equalities and the inequalities 
in 6) hold P-a.s. 


Proof 
1) For any BEG 


/ (aE(EIG) + bE(¢|G)) dP = af E(ElG) dP +6 i B(¢|G) dP 
B 


af cap+o | ¢ dP 


/ (a€ + b¢) dP 
B 


By uniqueness this proves the desired equality. 

2) This follows by putting A = 2 in (2.3). Also, 2) is a special case of 5) 
when H = {@, 2}. 

3) We first verify the result for € = 14, where A € G. In this case 


/ 1E(n|G) dP = / E(n\G) dP 
B ANB 


= if ndP 
ANB 


[ landP 
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for any B € G, which implies that 
14E(n|G) = E(1an|9) 


by uniqueness. In a similar way we obtain the result if € is a G-measurable step 


function, 
m 
E = ) ajla,, 
j=l 


where A; € G for j = 1,...,m. Finally, the result in the general case follows 
by approximating € by G-measurable step functions. 

4) Since € is independent of G, the random variables € and 1g are inde- 
pendent for any B € G. It follows by Proposition 1.1 (independent random 
variables are uncorrelated) that 


E(€)E(1z) 
= E(1p) 


[ édP, 


A E(€) dP 


which proves the assertion. 
5) By Definition 2.4 


[ E(é\G) aP = [ édP 


A E(€\H) dP = ip édP 


for every B € H. Because H C G it follows that 


| Balo)aP = i. B(é{H) aP 
B B 


for every B € G, and 


for every B € H. Applying Definition 2.4 once again, we obtain 
E(E(€|9)|H) = E(g|#). 


6) For any n we put 


An = {B (619) < -=}. 


Then A, € G. If € > 0 as., then 


o</ eap= | Bielg) dP <-=P (An), 
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which means that P(A,) = 0. Because 

(oe) 

{E (€|G) < 0} = [J An 

n=1 

it follows that 
P{E(§|G) < 0} =0, 
completing the proof. 0 
The next theorem, which will be stated without proof, involves the notion 

of a convex function, such as max (1, z) or e~*, for example. In this course the 


theorem will be used mainly for |x|, which is also a convex function. In general, 
we call a function y: R > R conver if for any z,y € R and any X € (0, 1] 


yp (Az +(1—A)y) < Av (x) + (1- A) ey). 


This condition means that the graph of » lies below the cord connecting the 
points with coordinates (x, y(x)) and (y,y (y)). 


Theorem 2.2 (Jensen's Inequality) 


Let y: R — R be a convex function and let € be an integrable random variable 
on a probability space (92,7, P) such that y (€) is also integrable. Then 


P(E (ElG)) SE(y(g)lg) as. 


for any o-field G on §2 contained in F. 


2.6 Various Exercises on Conditional 
Expectation 


Exercise 2.13 


Mrs. Jones has made a steak and kidney pie for her two sons. Eating more 
than a half of it will give indigestion to anyone. While she is away having tea 
with a neighbour, the older son helps himself to a piece of the pie. Then the 
younger son comes and has a piece of what is left by his brother. We assume 
that the size of each of the two pieces eaten by Mrs. Jones’ sons is random and 
uniformly distributed over what is currently available. What is the expected 
size of the remaining piece given that neither son gets indigestion? 
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Hint All possible outcomes can be represented by pairs of numbers, the portions of 
the pie consumed by the two sons. Therefore §2 can be chosen as a subset of the 
plane. Observe that the older son is restricted only by the size of the pie, while the 
younger one is restricted by what is left by his brother. This will determine the shape 
of §2. Next introduce a probability measure on §2 consistent with the conditions of 
the exercise. This can be done by means of a suitable density over §2. Now you are in 
a position to compute the probability that neither son will get indigestion. What is 
the corresponding subset of (2? Finally, define a random variable on §2 representing 
the portion of the pie left by the sons and compute the conditional expectation. 


Exercise 2.14 


As a probability space take 2 = [0,1) with the o-field of Borel sets and the 
Lebesgue measure on [0,1). Find E (€|n) if 


22 forO<2r<i 
— ise — jis 2° 
C(t) = 25", n (x) i for; <a<l. 


Hint What do events in a (7) look like? What do o (7)-measurable random variables 
look like? If you devise a neat way of describing these, it will make the task of 
finding EF (€|7) much easier. You will need to transform the integrals in condition 2) 
of Definition 2.3 to find a formula for the conditional expectation. 


Exercise 2.15 


Take §2 = [0, 1] with the o-field of Borel sets and P the Lebesgue measure on 
[0,1]. Let 
n(x) = (1 — 2) 


for x € [0,1]. Show that 


E(E|n)(x) = LD * SE =) 


for any z € [0, 1]. 
Hint Observe that n(x) = 7 (1 — 2). What does it tell you about the o-field generated 


by n? Is 5 (€ (w) + € (1 — z)) measurable with respect to this o-field? If so, it remains 
to verify condition 2) of Definition 2.3. 


Exercise 2.16 
Let €,7 be integrable random variables with joint density f¢ (x,y). Show that 


_ Jp® fen(2,9) de 


eASII Sr fejn(2,7) dx 


Hint Study the solutions to Exercises 2.7 and 2.8. 
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Remark 2.3 
If we put 
_ fen(Z,y) 
fein (xy) — fe (y) ’ 
where 


fn (y) = [ fenton) dx 


is the density of 7, then by the result in Exercise 2.16 


E (é|n) = [ 2 fen (ain) de. 


We call fen (zy) the conditional density of € given 7. 


Exercise 2.17 
Consider L? (F) = L* (Q,F, P) as a Hilbert space with scalar product 
L* (F) x L* (F) 3 (&,0) 4 E(€Q) ER. 


Show that if € is a random variable in L? (F) and G is a o-field contained in 
F, then E(€|G) is the orthogonal projection of € onto the subspace L? (G) in 
L? (F) consisting of G-measurable random variables. 


Hint Observe that condition 2) of Definition 2.4 means that €— E(€|G) is orthogonal 
(in the sense of the scalar product above) to the indicator function 1,4 for any A € G. 


2.7 Solutions 


Solution 2.1 
Since P(2) =1 and {,€dP = E(6), 


1 


E (€|22) = P(2) 


[ sap =2@. 


Solution 2.2 
By Definition 2.1 


E(1,4|B) = i fs dP 
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_ P(ANB) 
~ -P(B) 
P(A|B). 


Solution 2.3 


Since 7 is constant, it has only one value c € R, for which 
{n=ch =. 
Therefore E (€|n) is constant and equal to 
E(E|n)(w) = B(€| {n = c}) = E (€|2) = E (€) 


for each w € $2. The last equality has been verified in Exercise 2.1. 


Solution 2.4 


The indicator function 1g takes two values 1 and 0. The sets on which these 
values are taken are 


fie 1 a8. {lp =0}=2\B. 
Thus, for w € B 
E(lalla)(w) = E(14|B) = P(A|B), 
as in Exercise 2.2. Similarly, forw € 2\ B 
E(lalla)(w) = B(l4|2 \ B) = P(Al2 \ B). 


Solution 2.5 


First we observe that 


[ Beiwyar = f (sy [6 P) ap = | ¢ aP (2.4) 


for any event B. 

Since 7 is discrete, ii has countably many values y),y2,.... We can as- 
sume that these values are pairwise distinct, i.e. yi # y; if t # Jj. The sets 
{n = y1},{n = yo},... are then pairwise disjoint and they cover the whole 
space §2. Therefore, by (2.4) 


E(E(ln)) = / E(E|n) dP 


2 


= E =y,})dP 
2 = (él{n =n} 
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l| 

& 
— 
Yaa; 
— 


Solution 2.6 
First we need to describe the o-field o (7) generated by 7. Observe that 77 is 


symmetric about + 

n(x) =n(1—- 2) 
for any z € [0,1]. We claim that o (7) consists of all Borel sets A C [0,1] 
symmetric about x, i.e. such that 


ASA. (2.5) 
Indeed, if A is such a set, then A = {7 € B}, where 
B= {2r:2 € AN(0,$]} 


is a Borel set, so A € o(n). On the other hand, if A € a(n), then there is a 
Borel set B in R such that A= {n € B}. Then 


rEAS n(x) EB 
& n(l-z)EB 
& 1-2r€A, 


so A satisfies (2.5). 
We are ready to find FE (€|n). If it is to be o (n)-measurable, it must be 
symmetric about x, 1.€. 


E (|n) (x) = E(€|n) (1-2) 


for any x € [0,1]. For any A € o (7) we shall transform the integral below so 
as to make the integrand symmetric about 5: 


[ 2x az = [race f oras 
A A A 
= [ vars | (1—2)* dr 
A 1—A 
[ stare f ao) dx 
A A 
[ (22+(-2)) dx. 
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Figure 2.3. The graph of EF’ (€|7) in Exercise 2.6 


It follows that 
E (£|n) (x) = 2? + (1-2). 


The graphs of £, 7 and £ (€|n) are shown in Figure 2.3. 


Solution 2.7 


Since 
{7 € B}=(0,1]x B 


for any Borel set B, we have 


rem 6 7 J fe fenlou) dr dy 
I is eoee ix ay 
= J, (5+3) 4 


2+ 37 Jp Pa ff ee teal 
(x,y) dxrd 


and 


Because each event in o (7) is of the form {7 € B} for some Borel set B, this 
means that condition 2) of Definition 2.3 is satisfied. The random variable stan 
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is 0 (7)-measurable, so condition 1) holds too. It follows that 


2+ 37 
E (&|n) = 3467 


Solution 2.8 


We are looking for a Borel function F': R > R such that for any Borel set B 


i ar = i F (n) dP. (2.6) 
{nEB} {n€B} 


Then £ (€|n) = F'(n) by Definition 2.3. 
We shall transform both integrals above using the joint density fe, (x,y) 
in much the same way as in the solution to Exercise 2.7, except that here we 


do not know the exact form of F'(y). Namely, 


(y) 
| em = | [efenleva) de dy 
7 ol, (= (2 +9") te) dy 


and 


Ie? ee a [re y) fen(x, y) dx dy 
7 5 [,Fw ( (2 +9") ts dy 
- 5 [Fw (G+v) dy. 


Then, (2.6) will hold for any Borel set B if 


a+5y? — 34+ 6y’? 


F = See 
(y) Dey? 44 12y? 
It follows that ee 
+ 67 
(Eln) oar + ae 


Solution 2.9 


We are looking for a Borel function fF : R — R such that for any Borel set 


BCR 
/ €°dP = : F (n) dP. (2:7) 
{ne B} {ne B} 
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Then, by Definition 2.3 we shall have E (€*|7) = F (7). 

Let us transform both sides of (2.7). To do so we observe that the ran- 
dom variables € and 7 have uniform joint distribution over the unit disc 
Q = {(z,y): 2° +y? <1}, with density 


fen (x,y) ores 


as 


if c? + y? < 1, and fe, (x,y) = 0 otherwise. It follows that 


: €?dP = Lie fen (x,y) dx dy 
{ne B} BUJR 


1 Vind 

= 2 

= -| f x dx dy 
TV B —/l-y 2 


2 3/2 
ra y’)" dy 


[LPO fale) aeay 
few [aca 


- [ Fw) (1 —y?)'”? dy. 


as 


and 


J F(n) dP 
{nEB} 


II 


If (2.7) is to be satisfied for all Borel sets B, then 


F(y)= 3 (1-v’). 


This means that 
E (€*|n) (2, y) = F (n(2,y)) = F(y) = 
for any (z,y) in 2. 


Solution 2.10 


If G = {0,2}, then any constant random variable is G-measurable. Since 


[eap=B©= | Be aP 
[sap =0= [ B@ ap 


it FaAllaura that ACIC\) — AYL\ aes ae raniired 


and 
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Solution 2.11 


Because the trivial identity 


| cap = | ear 


holds for any A € G and € is G-measurable, it follows that E(E€|G) = € as. 


Solution 2.12 
By Definition 2.3 


[ea ap = | eaP 


since B € G. It follows that 


Recep) =~ [ E(E\G) dP 


Solution 2.13 


The whole pie will be represented by the interval [0,1]. Let x € [0,1] be the 
portion consumed by the older son. Then [0,1 — z] will be available to the 
younger one, who takes a portion of size y € [0,1 — z]. The set of all possible 
outcomes is 

Q ={(x,y):2,y>0,0+y <1. 


The event that neither of Mrs. Jones’ sons will get indigestion is 


1 
A={(aueQiny< 5} 


These sets are shown in Figure 2.4. If x is uniformly distributed over (0, 1] and 
y is uniformly distributed over [0,1 — x], then the probability measure P over 
f2 with density 


will describe the joint distribution of outcomes (x,y), see Figure 2.5. 
Now we are in a position to compute 


P(A) = [ fla,y)dedy 


1 1 
a f2 ] 
=| | dx dy 
0 0 l-—gz 
In /2 
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Figure 2.4. The sets {2 and A in Exer- 
cise 2.13 


Figure 2.5. The density f (x,y) in Exer- 
cise 2.13 


The random variable 


é(t,y)=1-a-y 
defined on 2 represents the size of the portion left by Mrs. Jones’ sons. Finally, 
we find that 


BUA) = pray [0-2 -w) feu)dedy 
- sal ps Gag aed 
_ 1—InV2 
Ind * 


Solution 2.14 


The o-field a(n) generated by 7 consists of sets of the form BU (B+ >) for 
some Borel set B C [0, 5). Thus, we are looking for a o (n)-measurable random 
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variable ¢ such that for each Borel set B C [0, $) 


tt) dz2= x) dz. 
| east Joona)’ a 


Then EF (€|n) = ¢ by Definition 2.3. 
Transforming the integral on the left-hand side, we obtain 


= [ arar+ f a(e+4)’ dr 
B B 


lI 
i) 
o 
——_ 
8 
i] 
+ 
— 
8 
+ 
vole 
—S 
w& 
—” 
Q. 
: 


For ¢ to be o (7)-measurable it must satisfy 
¢ (x) =C (r+ 5) (2.9) 


for each x € (0, 5). Then 


roan dP = [cw a+ fc dx 


l| 
wy 
=~ 
ae 
Qu. 
8 
+ 
wy 
a 
Qu. 
& 


| 
Lye) 
van 
& 
Q 
8 


If (2.8) is to hold for any Borel set B C 0, $), then 
C(x) = 2? + (x +4)” 


for each x € [0, 5). The values of ¢ (x) for z € [5,1) can be obtained from (2.9). 
It follows that 


Beemey=cay={ 2 Gt, for0<a2< 3, 


(a —1)° +2? for} <a<l. 


The graphs of €, 7 and E (&|n) are shown in Figure 2.6. 
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E(E|n) 


Figure 2.6. The graph of FE (€|7) in Exercise 2.14 


Solution 2.15 


Since n(z) = n(1 — 2), the o-field o(7) consists of Borel sets B C [0,1] such 


that 
B=1-B, 


where 1 — B = {1—2:2 € B}. For any such B 


[ sede = 5 [selae +5 | ete)ax 
= 5 cadres sf €(1 — 2) dx 
= 5 [| ee) )dxz + — Affe 
- {wee 14602) g 


Because 5 (€ (x) + €(1 — z)) is o (n)-measurable, it follows that 


B (é\n) (2) = SEE 2) 


Solution 2.16 
We are looking for a Borel function F'(y) such that 


/ cap = | F(n) dP 
(neB} (neB} 


for any Borel set B in R. Because F'(n) is o (7)-measurable and each event 
in o(n) can be written as {7 € B} for some Borel set B, this will mean that 


DMfrlevy — Mer 
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Let us transform the two integrals above using the joint density of € and 7: 


I cn = [ [ten eddy 
- I, (| = fen (x,y) ir) dy 


hnewy? Piaf: ies y) fen (z,y) dz dy 
= [Fw (ten (x,y) iz ae 


If these two integrals are to be equal for each Borel set B, then 


and 


Jat fen n ( dx 
F Te a 
= Sr fen (t,y) dx ~ 
It follows that 
E(é|n) = F(n) = Sa fen (2,0) de 


He Fen (r,n) dx - 


Solution 2.17 


We denote by ¢ the orthogonal projection of € onto the subspace L?(G) C 
L* (F) consisting of G-measurable random variables. Thus, € — ¢ is orthogonal 
to L? (G), that is, 
El(€ - ¢)y] =0 

for each y € L?(G). But for any A € G the indicator function 1,4 belongs to 
L? (G), so 

Ele =) Lal 0: 
Therefore 


| gap = B14) = Ea) = | caP 
A A 
for any A €G. This means that ¢ = E(€|G). 


a 


Martingales in Discrete Time 


3.1 Sequences of Random Variables 


A sequence £,&2,... of random variables is typically used as a mathematical 
model of the outcomes of a series of random phenomena, such as coin tosses 
or the value of the FTSE All-Share Index at the London Stock Exchange on 
consecutive business days. The random variables in such a sequence are indexed 
by whole numbers, which are customarily referred to as discrete time. It is 
important to understand that these whole numbers are not necessarily related 
to the physical time when the events modelled by the sequence actually occur. 
Discrete time is used to keep track of the order of events, which may or may 
not be evenly spaced in physical time. For example, the share index is recorded 
only on business days, but not on Saturdays, Sundays or any other holidays. 
Rather than tossing a coin repeatedly, we may as well toss 100 coins at a time 
and count the outcomes. 


Definition 3.1 


The sequence of numbers &€; (w) ,& (w),... for any fixed w € £2 is called a 


sample path. 


A sample path for a sequence of coin tosses is presented in Figure 3.1 (+1 
stands for heads and —1 for tails). Figure 3.2 shows the sample path of the 
FTSE All-Share Index up to 1997. Strictly speaking the pictures should con- 
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Figure 3.1. Sample path for a sequence of coin tosses 


Figure 3.2. Sample path representing the FTSE All-Share Index up to 1997 


sist of dots, representing the values €; (w) , 2 (w),... , but it is customary to 
connect them by a broken line for illustration purposes. 


3.2 Filtrations 


As the time n increases, so does our knowledge about what has happened in 
the past. This can be modelled by a filtration as defined below. 


Definition 3.2 
A sequence of o-fields Fi, F2,... on §? such that 


Fi C Fo C+: CF 
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is called a filtration. 


Here 7, represents our knowledge at time n. It contains all events A such 
that at time n it is possible to decide whether A has occurred or not. As n 
increases, there will be more such events A, i.e. the family 7, representing our 
knowledge will become larger. (The longer you live the wiser you become!) 


Example 3.1 
For a sequence £1, 2,... of coin tosses we take F,, to be the o-field generated 
by Cis 7 Cay 
Fx 0 (ijckew ba): 
Let 


A = {the first 5 tosses produce at least 2 heads}. 


At discrete time n = 5, i.e. once the coin has been tossed five times, it will be 
possible to decide whether A has occurred or not. This means that A € Fs. 
However, at n = 4 it is not always possible to tell if A has occurred or not. If 
the outcomes of the first four tosses are, say, 


tails, tails, heads, tails, 


then the event A remains undecided. We will have to toss the coin once more 
to see what happens. Therefore A ¢ Fy. 

This example illustrates another relevant issue. Suppose that the outcomes 
of the first four coin tosses are 


tails, heads, tails, heads. 


In this case it is possible to tell that A has occurred already at n = 4, whatever 
the outcome of the fifth toss will be. It does not mean, however, that A belongs 
to F4. The point is that for A to belong to F4 it must be possible to tell whether 
A has occurred or not after the first four tosses, no matter what the first four 
outcomes are. This is clearly not so in the example in hand. 


Exercise 3.1 


Let &),&2,... be a sequence of coin tosses and let ¥,, be the o-field generated 
by &1,.--,&n. For each of the following events find the smallest n such that the 
event belongs to Fp: 

A = {the first occurrence of heads is preceeded by no more than 10 tails}, 

B = {there is at least 1 head in the sequence ), &2,..-}, 

C' = {the first 100 tosses produce the same outcome}, 


N — Sthara ara nn mnra than 9 hoade and 9 taille amnnoa tha firet 5 trccacl 
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Hint To find the smallest element in a set of numbers you need to make sure that 
the set is non-empty in the first place. 


Suppose that £,,€),... is a sequence of random variables and F 1, Fo,... is 
a filtration. A priori, they may have nothing in common. However, in practice 
the filtration will usually contain the knowledge accumulated by observing the 
outcomes of the random sequence, as in Example 3.1. The condition in the def- 
inition below means that 7, contains everything that can be learned from the 


values of €:,...,€,- In general, it may and often does contain more information. 
Definition 3.3 
We say that a sequence of random variables ), £2,... is adapted to a filtration 


F1,F2,... if €p is Fp-measurable for each n = 1,2,.... 


Example 3.2 


If F, = o(£1,..-,&n) is the o-field generated by &,...,&,, then &1, £2,... is 
adapted to Fi, Fo,.... 


Exercise 3.2 


Show that 
a ae = OAC igxaeg tay) 


is the smallest filtration such that the sequence €1,2,... is adapted to Fj, 


Fo,.... That is to say, if G,,G2,... is another filtration such that 1, &,... is 
adapted to G1,G2,... , then F, C Gr for each n. 
Hint For o(€1,...,&n) to be contained in G, you need to show that &1,...,&, are 


Gn-measurable. 


3.3 Martingales 


The concept of a martingale has its origin in gambling, namely, it describes 
a fair game of chance, which will be discussed in detail in the next section. 
Similarly, the notions of submartingale and supermartingale defined below are 
related to favourable and unfavourable games of chance. Some aspects of gam- 
bling are inherent in the mathematics of finance, in particular, the theory of 
financial derivatives such as options. Not surprisingly, martingales play a cru- 
cial role there. In fact, martingales reach well beyond game theory and appear 
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in various areas of modern probability and stochastic analysis, notably, in dif- 
fusion theory. First of all, let us introduce the basic definitions and properties 
in the case of discrete time. 


Definition 3.4 


A sequence 1, €2,... of random variables is called a martingale with respect to 
a filtration F1, Fo,... if 


1) €, is integrable for each n = 1,2,... ; 
2) €1,€2,... is adapted to 71,Fo,...; 
3) B(;4i\Fn) = €y as: foreach t= 1, 2.3.0.4 


Example 3.3 


Let 7|,72,... be a sequence of independent integrable random variables such 
that E(nn) = 0 for all n = 1,2,.... We put 


Cy = ee as 
Fn = o(m,---s%n)- 


II 


Then &, is adapted to the filtration 7,, and it is integrable because 


B(\€nl) = E(lm +--+ ml) 
< E(lm|) +---+ E(|nnl) 
< OO. 


Moreover, 


E(En+1|Fn) as E(qn+1|Fn) + E(En|Fn) 
= E(nnti) + €n 
= €n; 
since 7)n41 is independent of F, (‘and independent condition drops out’) and 


€, is F,-measurable (‘taking out what is known’). This means that €, is a 
martingale with respect to Fn. 


Example 3.4 


Let € be an integrable random variable and let 71, 72,... be a filtration. We 


put 
E, = E(E\|Fn) 
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1OP S12 e's 
Then €, is 7,-measurable, 


which implies that 
E(\fnl) < B(E(lE| |Fn)) = E(él) < 0, 


and 


Ben sal Fn) = E(E(E|Fnii)|Fn) = E(€|Fn) =n, 


since F;, C Fn41 (the tower property of conditional expectation). Therefore €,, 
is a martingale with respect to Fy. 


Exercise 3.3 


Show that if €, is a martingale with respect to F,, then 


ES) = Blea) Ss 


Hint What is the expectation of E(&n41|Fn)? 


Exercise 3.4 


Suppose that €, is a martingale with respect to a filtration ¥,,. Show that €, 
is a martingale with respect to the filtration 


Ci =O Cisne yea). 


Hint Observe that Gn C F, and use the tower property of conditional expectation. 


Exercise 3.5 


Let £, be asymmetric random walk, that is, 


Ex =m +--+ Mn, 


where 71,72,... iS a sequence of independent identically distributed random 


variables such that : 


Pile = 1S Pie = l= 


(a sequence of coin tosses, for example). Show that €2 — n is a martingale with 
respect to the filtration 
Fn = o(m,. Sean): 
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Hint You want to transform E (E741 —(n+1) \Fn) to obtain 2 — n. Write 


ode eee (See Pee 
M41 + 2nn+16n + e? 


and observe that €, is #,-measurable, while 7,41 is independent of F,. To transform 
the conditional expectation you can ‘take out what is known’ and use the fact that 
‘an independent condition drops out’. Do not forget to verify that €2 — n is integrable 
and adapted to Fy. 


Exercise 3.6 


Let €, be a symmetric random walk and f,, the filtration defined in Exer- 
cise 3.5. Show that 
Cn = (—1)" cos(m€n) 


is a martingale with respect to Fy. 


Hint You want to transform E((—1)"*' cos(mén41)|Fn) to obtain (—1)” cos(7€n). 
Use a similar argument as in Exercise 3.5 to achieve this. But, first of all, make sure 
that ¢, is integrable and adapted to Fy. 


Definition 3.5 


We say that €1,€,... is a supermartingale (submartingale) with respect to a 
filtration Fy, Fo,... if 


1) €, is integrable for each n = 1,2,... ; 
2) €:,€2,...1s adapted to F1, Fo,... ; 
3) E(En4i|Fn) < &n (respectively, E(€:41|Fn) > En) as. for eachn = 1,2,.... 


Exercise 3.7 


Let €, be a sequence of square integrable random variables. Show that if €, is 
a martingale with respect to a filtration F,, then €2 is a submartingale with 
respect to the same filtration. 


Hint Use Jensen’s inequality with convex function y (zr) = 2?. 


3.4 Games of Chance 


Suppose that you take part in a game such as the roulette, for example. Let 
™m.M.... be a sequence of integrable random variables. where n. are vour 
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winnings (or losses) per unit stake in game n. If your stake in each game is 
one, then your total winnings after n games will be 


En = Mm +-°- +My. 


We take the filtration 
FO ise a) 


and also put 5 = 0 and Fg = {@, 2} for notational simplicity. 
If n — 1 rounds of the game have been played so far, your accumulated 
knowledge will be represented by the o-field #,_,. The game is fair if 


E(En|Fn-1) = En-1; 


that is, you expect that your fortune at step n will on average be the same as 
at step n — 1. The game will be favourable to you if 


E(En|Fn—1) 2 ones 


and unfavourable to you if 


E(En|Fn-1) < Casa 


for n = 1,2,... . This corresponds to €, being, respectively, a martingale, a 
submartingale, or a supermartingale with respect to F,, see Definitions 3.4 
and 3.5. 

Suppose that you can vary the stake to be a, in game n. (In particular, ay 
may be zero if you refrain from playing the nth game; it may even be negative if 
you own the casino and can accept other people’s bets.) When the time comes 
to decide your stake a, you will know the outcomes of the first n — 1 games. 
Therefore it is reasonable to assume that a, is ¥,;,_)-measurable, where F,,_ 
represents your knowledge accumulated up to and including game n — 1. In 
particular, since nothing is known before the first game, we take Fo = {0, 2}. 


Definition 3.6 


A gambling strategy a1,Q@2,... (with respect to a filtration F,,F 2,...) is a 
sequence of random variables such that a, is F,_,)-measurable for each n = 
1,2,..., where Fo = {@, 2}. (Outside the context of gambling such a sequence 
of random variables a, is called previsible.) 


If you follow a strategy a ,,Q@2,... , then your total winnings after n games 
will be 


Cn = QM +:°+ + Qn 
= ai (&; — fa) t-+°+ Qn (En — En-1)- 
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We also put Co = O for convenience. 

The following proposition has important consequences for gamblers. It 
means that a fair game will always turn into a fair one, no matter which gam- 
bling strategy is usec. If one is not in a position to wager negative sums of 
money (e.g. to run a casino), it will be impossible to turn an unfavourable 
game into a favourable one or vice versa. You cannot beat the system! The 
boundedness of the sequence a,, means that your available capital is bounded 
and so is your credit limit. 


Proposition 3.1 


Let a1,Q2,... be a gambling strategy. 
1) If a,,a2,... is a bounded sequence and &p, €1, €2,... is a martingale, then 
Co,¢1, ¢2,... is a martingale (a fair game turns into a fair one no matter 


what you do); 


2) Ifa,,a2,...is anon-negative bounded sequence and &o, £1, €2,... 1S a super- 
martingale, then Co, G1, G2,... is a supermartingale (an unfavourable game 
turns into an unfavourable one). 


3) If a,,a2,... is a non-negative bounded sequence and &p, &, 2,... is a sub- 
martingale, then C), (1, (2,... is a submartingale (a favourable game turns 
into a favourable one). 


Proof 


Because a, and (,-1 are F,_1;-measurable, we can take them out of the ex- 
pectation conditioned on F,_; (‘taking out what is known’, Proposition 2.4). 
Thus, we obtain 


E& (CnlFn-1) = AGA + An (Ex _ En-1) \F =4) 
Cn OnE (Cal Fas) = faci): 


If €, is a martingale, then 


On (E (EnlFn—1) — &n-1) = 0, 


which proves assertion 1). If £, is a supermartingale and a, > 0, then 


An (E (En|Fn—-1) = faa) < 0, 


proving assertion 2). Finally, assertion 3) follows because 
On (E (€n|Fn-1) — En-1) > 0 


if €£, is a submartingale and a, > 0. 0 
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3.5 Stopping Times 


In roulette and many other games of chance one usually has the option to quit 
at any time. The number of rounds played before quitting the game will be 
denoted by 7. It can be fixed, say, to be r = 10 if one decides in advance 
to stop playing after 10 rounds, no matter what happens. But in general the 
decision whether to quit or not will be made after each round depending on 
the knowledge accumulated so far. Therefore 7 is assumed to be a random 
variable with values in the set {1,2,...}U {oo}. Infinity is included to cover the 
theoretical possibility (and a dream scenario of some casinos) that the game 
never stops. At each step n one should be able to decide whether to stop playing 
or not, i.e. whether or not t = n. Therefore the event that + = n should be 
in the a-field F, representing our knowledge at time n. This gives rise to the 
following definition. 


Definition 3.7 


A random variable 7 with values in the set {1,2,...}U {oo} is called a stopping 
time (with respect to a filtration F,) if for each n = 1,2,... 


{r=n}€ Fp. 


Exercise 3.8 

Show that the following conditions are equivalent: 
1) {7 <n} € F,, for each n= 1,2,...: 

2) {r =n} € F, for each n = 1,2,.... 


Hint Can you express {7 <n} in terms of the events {7 = k}, where k = 1,...,n? 
Can you express {7 = 7} in terms of the events {7 < k}, where kK = 1,...,n? 


Example 3.5 (First hitting time) 


Suppose that a coin is tossed repeatedly and you win or lose £1, depending 
on which way it lands. Suppose that you start the game with, say, £5 in your 
pocket and decide to play until you have £10 or you lose everything. If €, is 
the amount you have at step n, then the time when you stop the game is 


7T=min{n: €, = 10 or O},"7 


and is called the first hitting time (of 10 or 0 by the random sequence &,). It 


MF Lutz nAAKAAN + tr tho Fltratian 
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Fn = 0 (E1,...,&). This is because 
{r=n}={0<& < 10}N---N{0 < &,-1 < 10} N {€, = 10 or O}. 


Now each of the sets on the right-hand side belongs to F,,, so their intersection 
does too. This proves that 
{P= n\ Ee F,, 


for each n, so T is a Stopping time. 


Exercise 3.9 


Let £, be a sequence of random variables adapted to a filtration 7, and let 
BCR be a Borel set. Show that the time of first entry of £, into B, 


7T=min{n: &, € B} 
is a stopping time. 


Hint Example 3.5 covers the case when B = (—o0, 0] U[10, oo). Extend the argument 
to an arbitrary Borel set B. 


Let £, be a sequence of random variables adapted to a filtration 7, and 
let + be a stopping time (with respect to the same filtration). Suppose that 
€, represents your winnings (or losses) after n rounds of a game. If you decide 
to quit after 7 rounds, then your total winnings will be €,-. In this case your 
winnings after n rounds will in fact be £;,n. Here a A b denotes the smaller of 


two numbers a and J, 
a\b=min(a,b). 


Definition 3.8 


We call €;,, the sequence stopped at rT. It is often denoted by €7. Thus, for 


each w € 22 
En (w) a EN COE 


Exercise 3.10 


Show that if €, is a sequence of random variables adapted to a filtration F,,, 
then so is the sequence €;,n. 


Hint For any Borel set B express {€-~an € B} in terms of the events {£, € B} and 
{r =k}, where k = 1,...,n. 
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We already know that it is impossible to turn a fair game into an unfair 
one, an unfavourable game into a favourable one, or vice versa using a gambling 
strategy. The next proposition shows that this cannot be achieved using a 
stopping time either (essentially, because stopping is also a gambling strategy). 


Proposition 3.2 


Let 7 be a stopping time. 


1) If €, is a martingale, then so is E;,n. 
2) If €, is a supermartingale, then so is €;,n. 


3) If €, is a submartingale, then so is €;,n. 


Proof 


This is in fact a consequence of Proposition 3.1. Given a stopping time 7, we 


put 
1 ifr >n, 
Aan = : es 
0 ifr < 7: 


We claim that a, is a gambling strategy (that is, a, is #,_,-measurable). This 
is because the inverse image {a, € B} of any Borel set B C R is equal to 


eC Feng 


if 0,1 ¢ B, or to 
20€ Fri 


if 0,1 € B, or to 

{an =1l}={r>n}={r>n-l1} € F,-1 
if 1 € B and 0 ¢ B, or to 

{an =O} ={r<n}={r<n-1}€F,-1 
if 1 ¢ B and O€ B. For this gambling strategy 

Eran = O1 (61 — 0) + °+> + On (En — En-1)- 


Therefore Proposition 3.1 implies assertions 1), 2) and 3) above. UO 
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Example 3.6 


(You could try to beat the system if you had unlimited capital and unlimited 
time.) The following gambling strategy is called ‘the martingale’. (Do not con- 
fuse this with the general definition of a martingale earlier in this section.) 
Suppose a coin is flipped repeatedly. Let us denote the outcomes by 71, 72,... , 
which can take values +1 (heads) or —1 (tails). You wager £1 on heads. If you 
win, you quit. If you lose, you double the stake and play again. If you win this 
time round, you quit. Otherwise you double the stake once more, and so on. 
Thus, your gambling strategy is 


a ad ifm = = mn-1 = tails, 
cs 0 otherwise. 


Let us put 
Cn =m + 2m +--+ + 2°" 


and consider the stopping time 
T =min{n: 7, = heads}. 


Then ¢;,n will be your winnings after n rounds. It is a martingale (check it!). 

It can be shown that P {tr < oo} = 1 (heads will eventualiy appear in the 
sequence 71,72,... with probability one). Therefore it makes sense to consider 
¢,. This would be your total winnings if you were able to continue to play the 
game no matter how long it takes for the first heads to appear. It would require 
unlimited time and capital. If you could afford these, you would be bound to 
win eventually because ¢,; = I identically, since 


5 et) ey deer Lea 


for any n. 


Exercise 3.11 


Show that if a gambler plays ‘the martingale’, his expected loss just before the 
ultimate win is infinite, that is, 


E (¢,-1) = —00. 


Hint What is the probability that the game will terminate at step n, i.e. that 7 = n? 
If r = n, what is ¢--1 equal to? This will give you all possible values of ¢--1 and 
their probabilities. Now compute the expectation of ¢,_}. 
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3.6 Optional Stopping Theorem 


If €, is a martingale, then, in particular, 


E(En) ama E(&1) 


for each n. Example 3.6 shows that E(€,) is not necessarily equal to E(&,) for 
a stopping time 7. However, if the equality 


E(&-) 7 E(&1) 


does hold, it can be very useful. The Optional Stopping Theorem provides 
sufficient conditions for this to happen. 


Theorem 3.1 (Optional Stopping Theorem) 


Let £, be a martingale and 7 a stopping time with respect to a filtration F, 
such that the following conditions hold: 


1) r<oas., 
2) €, is integrable, 
3) E(Enltrsn}) 70 as n> o0. 


Then 
E(€,) a E(&;). 


Proof 
Because 

Er = Eran + (€; = En) \yesn}, 
it follows that 


E(€é,) = E(E-an) Bs E(E-1t7>n}) = ElEnlgrsn}): 


Since €-,n is a martingale by Proposition 3.2, the first term on the right-hand 
side is equal to 

E(Eran) = E(é1). 
The last term tends to zero by assumption 3). The middle term 


OO 


E(E-1g,5n}) = Ss E(&1,,=x}) 


k=n+1 
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tends to zero as n — oo because the series 
E(€-) =), E(&1{,=n}) 
k=1 


is convergent by 2). It follows that E(€,) = E(€,), as required. 0 


Example 3.7 (Expectation of the first hitting time for a random walk) 


Let &, be a symmetric random walk as in Exercise 3.5 and let K be a positive 
integer. We define the first hitting time (of +K by &,) to be 


7 =min{n: |€,| = K}. 


By Exercise 3.9 7 is a stopping time. By Exercise 3.5 we know that €2 —n is a 
martingale. If the Optional Stopping Theorem can be applied, then 


E(é-—7r) = E(é?-1) =0. 
This allows us to find the expectation 
E(r) = E(&) = K?, 


since |£,| = K. 

Let us verify conditions 1)-3) of the Optional Stopping Theorem. 

1) We shall show that P{r = oo} = O. To this end we shall estimate 
P {rt >2Kn}. We can think of 2K n tosses of a coin as n sequences of 2K 
tosses. A necessary condition for 7 > 2K n is that no one of these n sequences 
contains heads only. Therefore 

1 nm 
P{r>2Kn}< (1- gx) +0 

as n — oo. Because {t > 2Kn} for n = 1,2,... is a contracting sequence of 
sets (ie. {7 > 2Kn} D {7 > 2K (n+ 1)}), it follows that 


Pir 60) = P(A (r> akn)} 


lm P{7r > 2Kn} =0, 
n—-+>0o 


lI 


completing the argument. 
2) We need to show that 


E (\& -t|) < 0. 
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Indeed, 


E(@) = PP {ran) 
ee 
=) >> (2Kn+k)P{r =2Kn+k} 
en 
> 2K (n +1) P{r > 2Kn} 


n=0 k=1 


[o.@) l n 
< 4K? > (n+) (1~ ze] 


n=0 


lA 


< OO; 


since the series )->”_, (n + 1) q” is convergent for any g € (—1,1). Here we have 
recycled the estimate for P {t > 2Kn} used in 2). Moreover, £2? = K?, so 


BC ¢|) <5 E(G) +2) 


= K* + E(r) 


< 00. 
3) Since €2 < K? on {7 > n}, 
E (€lgsnj) < K?P{r >n} 40 
as n — oo. Moreover, 
E (nlgr>ny) SE (t1fr>n}) > 0 


as n — oo. Convergence to 0 holds because E (rT) < co by 2) and {7 > n} is 
a contracting sequence of sets with intersection {7 = oo} of measure zero. It 
follows that 

E ((€2 = n) bisay) > 0, 


as required. 


Exercise 3.12 


Let £, be a symmetric random walk and Ff, the filtration defined in Exer- 
cise 3.5. Denote by 7 the smallest n such that |€,| = K as in Example 3.7. 
Verify that 

Cn = (-1)" cos[m (fn + K)] 
is a martingale (see Exercise 3.6). Then show that ¢, and 7 satisfy the con- 
ditions of the Optional Stopping Theorem and apply the theorem to find 


nes a\T1 
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Hint The equality ¢, = (—1)” is a key to computing E[(—1)’] with the aid of the 
Optimal Stopping Theorem. The first two conditions of this theorem are either obvious 
in the case in hand or have been verified elsewhere in this chapter. To make sure that 
condition 3) holds it may be helpful to show that 


|E(Grl¢r>n})| < P{r > n}. 


Use Jensen’s inequality with convex function y(z) = |xz| to estimate the left-hand 
side. Do not forget to verify that ¢, is a martingale in the first place. 


3.7 Solutions 


Solution 3.1 


A belongs to F11, but. not to F149. The smallest n is 11. 

B does not belong to F,, for any n. There is no smallest n such that B € Fy. 
C' belongs to Fio9, but not to F99. The smallest n is 100. 

Since D = @, it belongs to F, for each n = 1,2,.... Here the smallest n is 1. 


Solution 3.2 


Because the sequence of random variables €;, €2,... is adapted to the filtration 
G1,G2,... , it follows that £, is G,-measurable for each n. But 

Gi C Go Ciena 5) 
sO €;,...,& are G,-measurable for each n. As a consequence, 


Fr = o(£1,---,€n) C Gn 
for each n. 
Solution 3.3 
Taking the expectation on both sides of the equality 
En = E(En+ilFn), 


we obtain 


E(fn) = E(E(En4ilFn)) = E(&n41) 


for each n. This proves the claim. 


Solution 3.4 


The random variables €, are integrable because €, is a martingale with respect 
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to F,,. Since Gy, is the o-field generated by €,,...,&, it follows that €, is 
adapted to Gn. Finally, since Gn C Fn, 


én = E(En|Gn) 
E(E(En41|Fn)|Gn) 
E(En+11Gn) 


by the tower property of conditional expectation (Proposition 2.4). This proves 
that €, is a martingale with respect to Gp. 


Solution 3.5 


Because 
@—n=(mt---+M) —n 


is a function of 7,...,7n, it is measurable with respect to the o-field F, gen- 
erated by m,...,7n; i.e. €2 —n is adapted to F,,. Since 
|En| ps In +:-++,| < In| + +++ |r| =n, 


it follows that 
E(\€& —nl) < E(G{)t+n<n+n<o, 


so £2 — n is integrable for each n. Because 


oe =O eae. 


where €, and €2 are F,,-measurable and 7+ is independent of F,,, we can use 
Proposition 2.4 (‘taking out what is known’ and ‘independent condition drops 
out’) to obtain 


BE ily) E(n241|Fn) ae 2E(nn+1€n|Fn) a E(€2|Fn) 
E( nai) + 26rE (mi) +6 


SLs 


This implies that 
E(€414—"—-1|Fa) = @ — 1, 


so £2 — nis a martingale. 


Solution 3.6 


Being a function of £,, the random variable ¢, is #,-measurable for each n, 
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since €, is. Because |¢,| < 1, it is clear that C, is integrable. Because 7,41 is 
independent of 7, and €, is F¥,-measurable, it follows that 


E(GntilFn) = E ((-1)"*? cos(t (En + tn41)]|Fn) 
= (-1)"t'E (cos (7p) cos (77n41) |Fn) 
—(-1)"°E (sin (én) sin (7741) |Fn) 
= (=1)"*" 00s (Ey) B (cos (711m 41)) 
—(—1)"*' sin (7€,) E (sin (77n41)) 
(—1)" cos (mEn) 
= Cns 


lI 


using the formula 
cos (a + 8) = cosacos 8 — sinasin £. 


To compute F (cos (77n41)) and E (sin (77n41)) observe that nn4+1 = 1 or -1 
and 


cost = cos(—7) = —l, 
sina = sin(—7) =0. 


It follows that ¢, is a martingale with respect to the filtration Fy. 


Solution 3.7 


If €, is adapted to F,, then so is €2. Since &, = E (€n41|Fn) for each n and 
y (x) = x? is aconvex function, we can apply Jensen’s inequality (Theorem 2.2) 
to obtain 


P=a(e Galt Sela) 
for each n. This means that £2 is a submartingale with respect to F;. 
Solution 3.8 
1)>2). If + has property 1), then 
{7 <n} € Fn 


and 
{ren —1) € FyarC Fn, 


{r=n}={r<n}\{r<n-1} € Fy. 


2)=>1). If 7 has property 2), then 
{r=k}e€ Fic fF. 
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for each k = 1,...,n. Therefore 


{r<n}={7r=1}U---U{r=n} € F,. 


Solution 3.9 


If 
r=min{n:€, € B}, 


then for any n 
{r=n}= {8 ¢ BYN---N{En-1 E B}N {& € B}. 


Because B is a Borel set, each of the sets on the right-hand side belongs to the 
o-field F, = 0 (&,...,&,), and their intersection does too. This proves that 
{r =n} € F,, for each n, so 7 is a stopping time. 


Solution 3.10 
Let B C R be a Borel set. We can write 


{Eran € B} = {fn EB,r>n}vU U {E, E€B,r =k}, 
k=1 


where 
{fr € Br > n} = {& EB} N{r >n} Ee Fa 


and for each k = 1,...,n 
{& € B,r=k} ={& € B}N {rt =k} € Fe C Fn. 


It follows that for each n 


aren € B} Ee Pp: 


as required. 


Solution 3.11 


The probability that ‘the martingale’ terminates at step n is 
1 
Pyr=n}=— 


(n — 1 tails followed by heads at step n). Therefore 


ENG) =) GeaiP {ran} 
n=1 
= 1 
= ~9_..._— 9n-2\ 
= 2 (-1-2 ad 


\ 
| 
| 
| 

8 
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Solution 3.12 


The proof that ¢, is a martingale is almost the same as in Exercise 3.6. We 
need to verify that ¢, and 7 satisfy conditions 1)—3) of the Optional Stopping 
Theorem. 

Condition 1) has in fact been verified in Example 3.7. 

Condition 2) holds because |¢,| < 1, so E(|¢;|) < 1 < oo. 

To verify condition 3) observe that |¢,,| < 1 for all n, so 


\E(Cnlr>n})| < E(\Gnllyrsn}) 


< E(1g73n}) 
P{r >n}. 


The family of events {7 > n},n = 1,2,...is a contracting one with intersection 
{t = oo}. It follows that 


EGulgsny)| <P> nh} VP ir =o} 
as n —> oo. But 


P{r = oo} =0 


by 1), completing the proof. 
The Optional Stopping Theorem implies that 


E(¢,) = £(G) 
Because £, = K or —K, we have 
Cr = (-1)" cos[m(K + €-)] = (-1)’. 
Let us compute 
E(q) = -5 (cos [x (1 + K)] + cos [a (—1+ K)]) 
= cos(mK) =(-1)". 


It follows that 
E[(-1)"] = (-1)*. 


casein Sane omsnetee 
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Martingale Inequalities and Convergence 


Results on the convergence of martingales provide an insight into their structure 
and have a multitude of applications. They also provide an important inter- 
pretation of martingales. Namely, it turns out that a large class of martingales 
can be represented in the form 


fn = E (E|\Fn) ’ (4.1) 


where € = lim, €, is an integrable random variable and Fy, Fo,... is the filtra- 
tion generated by &,,&2,... , see Theorem 4.4 below. This makes it possible to 
think of €,&2,... as the results of a series of imperfect observations of some 
random quantity €. As n increases, the accumulated knowledge F, about € 
increases and €, becomes a better approximation, approaching the observed 
quantity € in the limit. 

We shall begin with a few classical inequalities for martingales, known as 
the Doob inequalities. They provide the tools we shall need to study the con- 
vergence of martingales and, later on, the properties of stochastic integrals. 
Then we shall present a classical result known as Doob’s Martingale Conver- 
gence Theorem, which provides the limit lim, &, of a martingale. However, 
Doob’s theorem has one inconvenient feature. It guarantees only that €, con- 
verges a.s., even though the limit is known to be an integrable random variable. 
However, to obtain (4.1) we need convergence in L!, which gives rise to a condi- 
tion called uniform integrability. This condition and its consequences, including 
(4.3), will be studied in the second section. Finally, as an example of an appli- 
cation reaching beyond the theory of martingales, we present an elegant proof 
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4.1 Doob’s Martingale Inequalities 


Proposition 4.1 (Doob’s maximal inequality) 
Suppose that £,,n € N, is a non-negative submartingale (with respect to a 


filtration F,,). Then for any \ > 0 


“= (max Sk 2 a) St (£1 (maraca &42a}) 


where 1, is the characteristic function of a set A. 


Proof 


We put &* = max €, for brevity. For A > 0 let us define 
an 


T=min{k<n:& >A}, 


if there is a k < n such that € > A, and 7 = n otherwise. Then 7 is a stopping 
time such that T < n a.s. Since €, is a submartingale, 


E(én) 2 E(€,). 


But 
E(€,) = E (& lter>a}) + E (&- Mer cay) - 


Observe that if 5 > A, then €, > A. Moreover, if &* < A, then tr = n, and so 
€, = €n. Therefore 


E(€n) > E(&-) > AP (& >A) + E (Exlyges cay), 
It follows that 
AP (&, >A) < E (fn) — E (Enles cn}) = E (Enlyex>n}) ; 


completing the proof. UO 


Theorem 4.1 (Doob’s maximal L* inequality) 
If £,,n € N, is a non-negative square integrable submartingale (with respect 
to a filtration F,,), then 


2 
< 4E|€,|?. (4.2) 


E nas Ek 
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Proof 


Put €° = maxg<n €&. By Exercise 1.9, Proposition 4.1, the Fubini theorem and 
finally the Cauchy—Schwarz inequality 


Ble? =2f wle>nasa! BlEtgegon) @ 


7% ( [... x wv) =2f & ( [ a aP 


2 | éageaP = 28 (G06) $2(Bléal*) (BIG) 


sil 
Dividing by (E ie ) , we get (4.2). 0 


The proof of Doob’s Convergence Theorem in the next section hinges on an 
inequality involving the number of upcrossings. 


Definition 4.1 


Given an adapted sequence of random variables €,, £2,... and two real numbers 
a < b, we define a gambling strategy a,,ao,... by putting 


Oy} = 0 
and for n = 1,2,... 


1 if a, = 0 and €, <a, 
A444 = 1 if an = 1 and €, < b, 
0 otherwise. 


It will be called the upcrossings strategy. Each k = 1,2,... such that a, = 1 
and ax41 = 0 will be called an upcrossing of the interval [a, 6]. The upcrossings 
form a (finite or infinite) increasing sequence 


Uy < Ug <7". 


The number of upcrossings made up to time n, that is, the largest k such that 
Up <n will be denoted by U,[a, 6] (we put U;,[a, b] = 0 if no such k exists). 


The meaning of the above definition is this. Initially, we refrain from playing 
the game and wait until €£, becomes less than a. As soon as this happens, we 
start playing unit stakes at each round of the game and continue until €, 
becomes greater than b. At this stage we refrain from playing again, wait until 
€,, becomes less than a, and so on. The strategy a, is defined in such a way 
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that @, = 0 whenever we refrain from playing the nth game, and a, = 1 
otherwise. During each run of consecutive games with a, = 1 the process Ey, 
crosses the interval [a, b], starting below a and finishing above b. This is what 
is meant by an upcrossing. Observe that each upcrossing will increase our total 
winnings by at least b — a. For convenience, we identify each upcrossing with 
its last step k, such that a, = 1 and a,4; = 0. A typical sample path of the 
upcrossings strategy is shown in Figure 4.1. 


Exercise 4.1 


Verify that the upcrossings strategy a, is indeed a gambling strategy. 


Hint You want to prove that a, is F,-1-measurable for each n. Since the upcrossings 
strategy is defined by induction, a proof by induction on n may be your best bet. 


Lemma 4.1 (Upcrossings Inequality) 
If €;,€2,... is a supermartingale and a < b, then 
(b — a) E(U,[a, b]) < E((E: — @) ). 


By z~ we denote the negative part of a real number g, i.e. z~ = max {0, —z}. 


Proof 


Let 
Cn = O1 (£1 — 0) +°-- + On (En — En-1) 


be the total winnings at step n = 1,2,... if the upcrossings strategy is followed, 

see Figure 4.1. It will be convenient to put (9 = 0. By Proposition 3.1 (one 

cannot beat the system using a gambling strategy) ¢, is a supermartingale. 
Let us fix an n and put k = U,[a, }], so that 


O< uy <Ug<--: <upcn 
Clearly, each upcrossing increases the total winnings by b — a, 
Cui — Guia 24-4 
fori =1,...,k. (We put up = 0 for simplicity.) Moreover, 
Cn — Guz. 2 — (En — @) 


It follows that 
Cn > (b- a)U,,[a, }] (Gy) 2 
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Figure 4.1. Typical paths of £n, an and Cn; upcrossings are indicated by bold lines 


taking the expectation on both sides, we get 
E (Cn) 2 (b— a) E(Up[a, b]) — E((En — a) ). 
But Cn is a supermartingale, so 
0=E(G) 2 En); 


which proves the Upcrossings Inequality. 0 


4.2 Doob’s Martingale Convergence Theorem 


Theorem 4.2 (Doob’s Martingale Convergence Theorem) 


Suppose that &,,... is a supermartingale (with respect to a filtration 
F,,F2,...) such that 
sup E (|En|) < oo. 
n 


Then there is an integrable random variable € such that 


lim ;=€ as. 
N—+ Co 
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Remark 4.1 


In particular, the theorem is valid for martingales because every martingale is a 
supermartingale. It is also valid for submartingales, since €, is a submartingale 
if and only if —&, is a supermartingale. 


Remark 4.2 


Observe that even though all the €, as well as the limit € are integrable random 
variables, it is claimed only that €, trends to € a.s. Note that no convergence 
in L! is asserted. 


Proof (of Doob’s Martingale Convergence Theorem) 


By the Upcrossings Inequality 


B(G-9") Metal 


< 
E(U,{a, 6) < b-—a - b-a 


where 
M = sup E (|én|) < oo. 


Since U,,[a, 6] is a non-decreasing sequence, it follows that 


oe M + |al 
E (lim U,,(a, b)) = jim £ (nla, bl) < pe 
This implies that 
PY lim U,,[a, b] < 00 } =, 


for any a < b. Since the set of all pairs of rational numbers a < 6 is countable, 
the event 
A= f) { jim, U,,[a, b] < 00 } (4.3) 
a<b rational 


has probability 1. (The intersection of countably many events has probability 1 
if each of these events has probability 1.) 
We claim that the sequence €, converges a.s. to a limit €. Consider the set 


B = {liminf €, < limsup&,} C 2 


on which the sequence €, fails to converge. Then for any w € B there are 
rational numbers a, b such that 


lim inf €,(w) <a <b < limsup &,(w), 


ee ee ee ee 


| 
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implying that limn+o. Un[a, b](w) = oo. This means that B and the event A in 
(4.3) are disjoint, so P(B) = 0, since P(A) = 1, which proves the claim. 

It remains to show that the limit € is an integrable random variable. By 
Fatou’s lemma 


F (\€1) 


II 


E (tim inf nl) 
lim inf E (|&,|) 


IA 


\ 


sup E (||) < 00. 
This completes the proof. O 


Exercise 4.2 


| Show that if €, is a non-negative supermartingale, then it converges a.s. to an 


\ integrable random variable. 


int To apply Doob’s Theorem all you need to verify is that the sequence €, is 
bounded in L’, i.e. the supremum of F'(|€,|) is less than oo. 


NS 


4.3 Uniform Integrability and LZ! Convergence 
of Martingales 


The conditions of Doob’s theorem imply pointwise (a.s.) convergence of martin- 
gales. In this section we shall study convergence in L!. To this end we introduce 
a stronger condition called uniform integrability. Proposition 4.2 shows that it 
is a necessary condition for L' convergence. In Theorem 4.2 we prove that uni- 
form integrability is in fact sufficient for a martingale to converge in L}. This 
enables us to show that each integrable martingale is of the form EF (€|F,,). As 
an application we give a martingale proof of Kolmogorov’s 0-1 law. 


Exercise 4.3 


Show that a random variable € is integrable if and only if for every e > 0 there 
exists an M > 0 such that 


i; \€| dP <e. 
{|é|>M} 


Hint Split Q into two sets: {|€| > M} and {|€| < M}. The integrals of |€| over these 
sets must add up to E'(|é|). As M increases, one of the integrals increases, while the 
other one decreases. Investigate their limits as M —> oo. 
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Thus, for any sequence €, of integrable random variables and any € > 0 
there is a sequence of numbers M,, > 0 such that 


i; lEn| dP <e. 
{lén|>Mn} 


If the M,, are independent of n, then we say that the sequence €, is uniformly 
integrable. 


Definition 4.2 


A sequence €,&,... of random variables is called uniformly integrable if for 
every € > 0 there exists an M > 0 such that 


| lén| dP <eé 
{|€n|>M} 


for alln = 1,2,.... 

Exercise 4.4 

Let 2 = [0,1] with the o-field of Borel sets and Lebesgue measure. Take 
fn = M1io,2). 

Show that the sequence &j, 2,... is not uniformly integrable. 


Hint What is the integral of €, over {fn > M} ifn > M? 


Proposition 4.2 


Uniform integrability is a necessary condition for a sequence £1, £2,... of inte- 
grable random variables to converge in L?. 


Lemma 4.2 


If € is integrable, then for every e > 0 there is a 6 > O such that 


P(A) <5 | \glaP<e 
A 


Proof (of Lemma 4.2) 
Let € > 0. Since € is integrable, by Exercise 4.3 there is an M > 0 such that 


[ (ede < &. 


| 
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Now 


i. el dP 


[taps fe clap 
AN{|é| <M} AN{|€|>M} 


< | mars [ |é| dP 
A {|€|>M} 


< MP(A) + 


bol m 


Let 6 = 557. Then 
P(A) <5 => | If|dP <e, 
A 


as required. O 


Exercise 4.5 


Let € be an integrable random variable and Fj, Fo,... a filtration. Show that 
FE (€\|F,) is a uniformly integrable martingale. 


Hint Use Lemma 4.2. 


Proof (of Proposition 4.2) 


Suppose that £, > € in L}, i.e. E|E, — €| > 0. We take any e > 0. There is an 
integer N such that ‘ 
tee Dae EG or S| 5 


By Lemma 4.2 there is a 6 > 0 such that 
P(A) < b= | pape. 
A 2 
Taking a smaller 6 > 0 if necessary, we also have 
P(A) <é= | [fn] dP <e forn=1,...,N. 
A 
We claim that there is an M > 0 such that 


P {|&,| > M} <6 


for all n. Indeed, since 


B (\énl) > f 


fn] dP > MP {|En| > M}, 
{lén|>M} 


it suffices to take 1 
M = =sup & (|é,|). 
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‘Because the sequence €, converges in L’, it is bounded in L!, so the supremum 


s < 00.) 
Now, since P {|&,| > M} <6, 


| én | aP < | lap + | le, — €|dP 
{|fn|>M} {lEn|>M} {lEn|>M} 


I] dP + E(\E, — €|) 
ie 


IA 
se — 


or any n > N and 


i lEn|dP <e 
{lén|>M} 


or any n = 1,...,.N, completing the proof. U 


=xercise 4.6 


show that a uniformly integrable sequence of random variables is bounded in 
ut, ie. 
sup & (\En\) < ©. 
nr 


fint Write E ({€,|) as the sum of the integrals of |€,| over {|E,| > M} and {|€,| < M}. 


Exercise 4.6 implies that each uniformly integrable martingale satisfies the 
onditions of Doob’s theorem. Therefore it converges a.s. to an integrable ran- 
lom variable. We shall show that in fact it converges in L?. 


[heorem 4.3 


ivery uniformly integrable supermartingale (submartingale) £,, converges in L?. 


roof 


3y Exercise 4.6 the sequence €,, is bounded in L!, so it satisfies the conditions 
f Theorem 4.2 (Doob’s Martingale Convergence Theorem). Therefore, there is 
n integrable random variable € such that £,; — € a.s. We can assume without 
dss of generality that € = 0 (since €, — € can be taken in place of £,). That is 
O say, 
P {lim &y = o} =] 
n 
t follows that £, — 0 in probability, i.e. for any e > 0 


Pilflyselon 


eee ee 
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4. Martingale Inequalities and Convergence 77 


as n — oo. This is because by Fatou’s lemma 
limsup P {|£,| >e} < P (iim sup {|£,| > :}) 


< P (2 \ {lim én = o}) 
=: 


Let ¢ > 0. By uniform integrability there is an M > 0 such that 


| lEnldP < = 
{lén|>M} 3 


for all n. Since £,, > 0 in probability, there is an integer N such that ifn > N, 


then 
E 


P{lEn > =} < 3M 


We can assume without loss of generality that M > 3. Then 


| lén| aP + | lén| dP 
{|En|>M} {M>|én|>£} 


+ i soteer Ol? 
< 5+MP{iél > 5} + sP {5 > lel} 
€ 


E (nl) 


l 


for alln > N. This proves that E (|€,|) > 0, that is, & 40 in L!.O 


Theorem 4.4 
Let €, be a uniformly integrable martingale. Then 
En =E (E|Fn) ’ 


where € = limp €,, is the limit of €, in L1 and Fy = 0 (&,...,€n) is the filtration 
generated by £p. 


Proof 


For anym>n 


E (€m|Fn) =n, 


[ gmap =f ar, 


i.e. for any AE F, 


78 Basic Stochastic Processes 


Let n be an arbitrary integer and let A € F,,. For anym>n 


[@-9a| = [n-0 aP 
JA A 

< [ie é 

< E(|Em — €|) 30 


as m — oo. It follows that 


| fap = | ear 


for any A € Fn, so€, = E(E|F,). O 


Exercise 4.7 


Show that if €, is a martingale and , — a in L' for some a € R, then 
En, =a a.s. for each n. 


Hint Apply Theorem 4.4. 


Theorem 4.5 (Kolmogorov’s 0-1 Law) 


Let 71,72,... be a sequence of independent random variables. We define the 
tail o-field 
ESP ove O33: 3 
where Jn = 0 (Nn, 7n41,---). Then 
P(A)=0Oorl 
for any — 
Proof 
Take any A € 7 and define 
én = E(lalFn), 
where F, = o(m,.--,;%). By Exercise 4.5 €, is a uniformly integrable mar- 


tingale, so €,, > € in L'. By Theorem 4.4 
E (&\Fn) = E (lalFn) 


for all n. Both = lim, €, and 1,4 are measurable with respect to the o-field 


TC —Alm. m-. \ 
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The family G consisting of all sets B € ¥ such that [, €dP = [, ladP is ao- 
field containing #,; UF, U---.Asaresult, G contains the o-field 7. generated 
by the family 7, UF. U--:. By Lemma 2.1 it follows that € = 1,4 as. 

Since 7, is a sequence of independent random variables, the o-fields F,, 
and 7,4, are independent. Because 7 C 7nii, the o-fields 7, and 7 are 
independent. Being 7-measurable, 14 is therefore independent of F, for any 
n. This means that 


én = E(la|Fn) = E(1a) = P(A) as. 


Therefore the limit limnp_,o €) = € is also constant and equal to P(A) a.s. This 
means that P(A) =1,4 a.s.,so P(A) =Oorl.O 


Exercise 4.8 


Show that if A, € o (€,) for each n, then the events 


limsup A, = () U Aj 


g21i2) 
and 
lim inf An= U () Aj 
belong to the tail o-field 7. 
Hint You need to write limsup, A, and liminf, A, in terms of Ax, Ax4i,... for any 


k, that is, to show that limsup, A, and liminf, An will not be affected if any finite 
number of sets are removed from the sequence Aj, Ao,... . 


Exercise 4.9 


Use Kolmogorov’s 0-1 law to show that in a sequence of coin tosses there are 
a.s. infinitely many heads. 


Hint Show that the event 
{mi,72,... contains infinitely many heads} 


belongs to the o-field 7. Can the probability of this event be 0? The probability of 
the event 
{,72,... contains infinitely many tails} 


should be the same. Can both probabilities be equal to 0? Can you simultaneously 
have finitely many heads and finitely many tails in the sequence 71, 72,... ? 
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4.4 Solutions 


Solution 4.1 


Because a, = 0 is constant, it is Fo = {0, 2}-measurable. Suppose that ap is 
F,—1-measurable for some n = 1,2,... . Then 


{an =0,€& <a} U {an =1,6, < bh} EF, 
because F,_) C Fy and €, is Fn-measurable. This means that 


Anti = Ltan=0,£n<a}U{an=lén <b} 
is F,-measurable. By induction it follows that a, is F,_1-measurable for each 
nm=1,2,...,80 @1,Q@2,... is a gambling strategy. 
Solution 4.2 


For a non-negative supermartingale 


sup E (én) = sup E (én) < E(&) = E(|&|) < ©, 


since 


for each n = 1,2,... . Thus Doob’s Martingale Convergence Theorem implies 
that €, converges a.s. to an integrable limit. 


Solution 4.3 
Necessity. Suppose that € is integrable. It follows that 


P {|€| < co} = 1. 


The sequence of random variables |€| 1,)¢) 4} indexed by M = 1, 2,...is mono- 
tone and 

[E| Lyey> uy \, 0 as M - 00 
on the set {|&| < oo}, i.e. a.s. By the monotone convergence theorem for inte- 
grals 


[ If] dP \,0 as M - oo. 
J t\g1>M) 


It follows that for every € > 0 there exists an M > 0 such that 


/ |€| dP <e. 
{|€|>M} 


4. Martingale Inequalities and Convergence 81 


Sufficiency. Take « = 1. There exists an M > 0 such that 


i || dP <1. 
{|€|> 1} 


Then 
B(\él) = 7, lel aP 


| el dP + i lel aP 
{|€| >A} {\€|<M} 
1+ MP{\e| <M} 


1+ M <oo. 


IN A 


Solution 4.4 
For any M >0 and any n> M we have 
(0, +) — {En > M}, 


i, dP = | PL 
{En >M} (0,4) 


This means that there is no M > 0 such that for all n 


i: é,dP <4. 
{En >M} 2 


The sequence €, is not uniformly integrable. 


SO 


Solution 4.5 


In Example 3.4 it was verified that £, = E(€|7,) is a martingale. Let e > 0. 
By Lemma 4.2 there is a 6 > 0 such that 


P(A) < 5 | |€|dP <e. 
A 
By Jensen’s neGuAliEy {eI < E(\€||Fn) a.s., so 
E(\€|) >/E(l&n|) > 
eal) > 


If we take M > E({€|)/6, then 
P{\&,| > M} <6. 
Since {|€,| > M} € Fp, it follows that 


i énldP < / E(|€||Fa) 4P = 7 eldP <e, 
{|En|>M} {|En|>M} {lEn|>M} 


vroving that & = E(¢|F.,) is a uniformly integrable sequence. °° ' 


In| dP > MP{|fn| > M}. 


n|2 
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Solution 4.6 


Because €, is a uniformly integrable sequence, there is an M > 0 such that for 


all n 
i: lénldP <1. 
{lénl>M} 


It follows that 


E (|€nl) 


tI 


/ IE, | dP +f \€,| dP 
{l€n|>M} {lEn|<M} 


<1+MP{|&| <M} 
<1+M<o 


for all n, proving that. €, is a bounded sequence in L?. 


Solution 4.7 

By Theorem 4.4, €, = E(a|F,,) a.s. But E (a|F,,) = a a.s., which proves that 
En =Qa.s. 

Solution 4.8 


Observe that 


limsup A, = () U A; 


J2kK i125 


for any k. Since 


Aj € 0(&:) C Te 
for every i > k, it follows that 


lim sup Ay, = () UJ A; € Ty 


j>kidj 


for every k. Therefore 
sup A, € 7. 
nm 


The argument for liminf, A, is similar. 


Solution 4.9 


Let 71,72,... be a sequence of coin tosses, i.e. independent random variables 
with values +1, —1 (heads or tails) taken with probability 5 each. Consider the 
following event: 


A = {m,72,... contains infinitely many heads}. 
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This event belongs to the tail o-field 7 because 
A = limsup Ay, 
where 
An = {nn = heads} € a (mn) 


(see Exercise 4.8). Thus, by Kolmogorov’s 0-1 law P(A) = 0 or 1. However, it 
cannot be 0 because the event 


B = {m,%,... contains infinitely many tails} 


has the same probability by symmetry and (2 = AUB (there must be infinitely 
many heads or infinitely many tails). 


i) 


Markov Chains 


This chapter is concerned with an interesting class of sequences of random 
variables taking values in a finite or countable set, called the state space, and 
satisfying the so-called Markov property. One of the simplest examples is pro- 
vided by a symmetric random walk €, with values in the set of integers Z. If £, 
is equal to some 7 € Z at time n, then in the next time instance n+1 it will jump 
either to i+ 1, with probability $, or to i — 1, also with probability $. What 
makes this model interesting is that the value of ,,1 at time n + 1 depends 
on the past only through the value at time n. This is the Markov property 
characterizing Markov chains. There are numerous examples of Markov chains, 
with a multitude of applications. 

From the mathematical point of view, Markov chains are both simple and 
difficult. Their definition and basic properties do not involve any complicated 
notions or sophisticated mathematics. Yet, any deeper understanding of Markov 
chains requires quite advanced tools. For example, this is so for problems related 
to the long-time behaviour of Markov processes. In this chapter we shall try 
to maintain a balance between the accessibility of exposition and the depth of 
mathematical results. Various concepts will be introduced. In particular, we 
shall discuss the classification of states and its relevance to the asymptotic 
behaviour of transition probabilities. This will turn out to be closely linked to 
ergodicity and the existence and uniqueness of invariant measures. 
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5.1 First Examples and Definitions 


Example 5.1 


In some homes the use of the telephone can become quite a sensitive issue. 
Suppose that if the phone is free during some period of time, say the nth 
minute, then with probability p, where 0 < p < 1, it will be busy during the 
next minute. If the phone has been busy during the nth minute, it will become 
free during the next minute with probability g, where 0 < q < 1. Assume that 
the phone is free in the Oth minute. We would like to answer the following two 


questions. 

1) What is the probability z, that the telephone will be free in the nth 

minute? 

2) What is limn_. tn, if it exists? 

Denote by A, the event that the phone is free during the nth minute and let 
By, = 22\ An be its complement, i.e. the event that the phone is busy during 
the nth minute. The conditions of the example give us 
D, (5.1) 
q; (5.2) 
We also assume that P(Ao) = 1, ie. zo = 1. Using this notation, we have 
Ln = P(A,). Then the total probability formula, see Exercise 1.10, together 
with (5.1)-(5.2) imply that 

In4+1 = P(An41) 

P(An41|An)P(An) + P(Anti|Bn)P(Bn) 

(1 —p)tn+q(1—-tn) =9+(1—-p- Q)rn. (5.3) 


P(Bny+1 |An) 
P(An+1|Bn) 


It’s a bit tricky to find an explicit formula for z,. To do so we suppose first 
that the sequence {z,,} is convergent, i.e. 


lim: -¢4-=.2. (5.4) 
n— oo 


The elementary properties of limits and equation (5.3), ie. Zn41 =q+(1—p- 
q)Zn, yield 
r=qt+(l1—p-gq)z. (5.5) 


The unique solution to the last equation is 


qd 
f= —. 5.6 
ar (5.6) 
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In particular, 


qd qd 
neo ( jer (5.7) 
Subtracting (5.7) from (5.3), we infer that 
q q . 
i Se ape pes). 5.8 
aaa ae (1—p a)(2 tt (5.8) 


Thus, {rn — ee is a geometric sequence and therefore, for all n € N, 


q q 
L - t= (-p—4)" (mo - —L). 
" q+p q+p 


Hence, by taking into account the initial condition zp = 1, we have 


qd qd n 
In = —~—+ (2% -——](1-p-@ 
q+p (% ee) 
q , Pp 
= —— + ——(1l-p- gq)”. 5.9) 
cae Fags Pp — 4) ( 


Let us point out that although we have used the assumption (5.4) to derive 
(5.8), the proof of the latter is now complete. Indeed, having proven (5.9), we 
can show that the assumption (5.4) is indeed satisfied. This is because the 
conditions 0 < p,q < 1 imply that |1 — p— q| <1, and so (l-—p-—gq)" 40 as 
n — oo. Thus, (5.4) holds. This provides an answer to the second part of the 
example, i.e. limn.o Zn = Pore 


The following exercise is a modification of the last example. 


Exercise 5.1 


In the framework of Example 5.1, let y, denote the probability that the tele- 
phone is busy in the nth minute. Supposing that yo = 1, find an explicit formula 
for yn and, if it exists, lim,_,. Yn. 


Hint This exercise can be solved directly by repeating the above argument, or indi- 


rectly by using some of the results in Example 5.1. 


Remark 5.1 


The formulae (5.3) and (5.64) can be written collectively in a compact form by 
using vector and matrix notation. First of all, since t, + yn = 1, we get 


Inti = (1—p)in + Gyn; 
PIn + (1 — q)Yyn- 


Yn+1 
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Hence, the matrix version takes the form 
Me Sede) 
Pp l—q Yn 
The situation described in Example 5.1 is quite typical. Often the proba- 
bility of a certain event at time n+ 1 depends only on what happens at time 


n, but not further into the past. Example 5.1 provides us with a simple case of 
a Markov chain. See also the following definition and exercises. 


| In+1 
Yn+1 


Definition 5.1 


Suppose that S is a finite or a countable set. Suppose also that a probability 
space (2, F, P) is given. An S-valued sequence of random variables €,, n € N, 
is called an S-valued Markov chatn or a Markov chain on S if for all n € N and 
alls eS 

P(En41 = 8l€0,---,€n) = PlEnti = sl€n). (5.10) 
Here P(£n41 = s|€,) is the conditional probability of the event {€,41 = s} with 
respect to random variable €,,, or equivalently, with respect to the o-field a(€,) 
generated by €,. Similarly, P(€n41 = s|€0,.-.,&n) is the conditional probability 
of {€n41 = s} with respect to the o-field a(£0,:--,&n) generated by the random 


variables &,---,&n.- 
Property (5.10) will usually be referred to as the Markov property of the 


Markov chain £,,n € N. The set S is called the state space and the elements 
of S are called states. 


Proposition 5.1 


The model in Example 5.1] and Exercise 5.1 is a Markov chain. 


Proof 


Let S = {0,1}, where 0 and 1 represent the states of the phone being free or 
busy. First we need to construct an appropriate probability space. Let 2 be 
the set of all sequences wo,w ,... with values in S. Let fo be any probability 
measure on S. For example, fp = 69 corresponds to the case when the phone 
is free at time 0. We shall define P by induction. For any S-valued sequence 
So, S1,--. we put 

P({w € 2: wo = 8o}) = pol({so}) (5.11) 


and 


BiG 62 eis er ee EA) (5.12) 


b 
. 
B 
bE 
Bi 
Pp 
e 
E 
B 
i 
a 
f 
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= P(Sn+1|8n)P({w Ef: Wi = Si, 1= 0, aah ,n}), 
where p(s|r) are the entries of the 2 x 2 matrix 
| p(0|0)  p(0/1) | 2 L=p. | | 
p(1|0)  p(1|1) pe eg: 

It seems reasonable to expect P to be a probability measure (with respect to 
the trivial o-field of all subsets of (2). Take this for granted, check only that 
PQ) =. 

How would you define the process €,, n € N? We shall do it in the standard 


way, 1.€. 
Exn(w) =Wn, WEN. (5.13) 


First we shall show that the transition probabilities of £, are what they should 
be, i.e. 
P(fnt1 = 1 = 0) =p, (9:16) 
P(En+1 = Ol. =1) =¢. (5.15) 


The definition of conditional probability yields 


PGagy = 16,0) = SS a 


Next, the definition of P gives 

P(€n41 = 1, &n = 0) 

= P({w me 0 a0 O,Wn41 = 1}) 

= S” P({w € 2:0; = 8,1 =0,---,n-—1, wp = 0,Wn41 = 1}) 
$0,°7;Sn-1€S 

Ss pP({w € 2: uw; = 8;,1=0,---,n-1, wp = 0}) 
$0,°';8n-1ES 


pP(En = 0), 


by (5.12). We have proven (5.14). Moreover, (5.15) follows by the same argu- 
ment. A similar line of reasoning shows that €, is indeed a Markov chain. For 


this we need to verify that for any n € N and any $9, 81,°--,$n41 €S 
P(€n+1 = 8n4il60 = $0,°°*s én = Sn) = PlEngi = Sn4il€n = Sn). 
We have 
PGs sti Senta s) 


P({w € 2:0; = 83,1 =0,---,n+1}) 
P(8n41|8n)P({w EN: Wi ne = 0,---,n}) 
€ 


P(Sn41|$n)P (Eo = 80,°°'° 5 On\= Sn) 
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which, in view of the definition of conditional probability, gives 


Pena = Sas Co= 80527 ion = Sn) =P Crlsa). 


On the other hand, by an easy generalization of (5.14) and (5.15) 
PLE 4 = $a14 Gn = Sn) = P(Sn41\|8n), 
which proves (5.10). O 


In Example 5.1 the transition probabilities from state i to state 7 do not 
depend on the time n. This is an important class of Markov chains. 


Definition 5.2 


An S-valued Markov chain &,, n € N, is called ttme-homogeneous or homoge- 
neous if for alln € N and alli,j ES 


P(Enti = Jif = %) = P(f = gl€o = 2). (5.16) 


The number P(€; = j|& = 2) is denoted by p(j|i) and called the transition 
probability from state i to state j. The matrix P = [p(j|t)]; ;-5 is called the 
transition matrix of the chain €,. 


Exercise 5.2 


In the discussion so far we have seen an example of a transition matrix, P = 

| l=p 4 
p 1-q 

to 1. Prove that this is true in general. 


| . Obviously the sum of the entries in each column of P is equal 


Hint Remember that P(2|A) = 1 for any event A. 


Definition 5.3 
A = [ajilijesg is called a stochastic matrix if 

1) aj; > 0, for all 2,7 € S; 

2) the sum of the entries in each column is 1, i.e. jes aj, = 1 for anyi €S. 


A is called a double stochastic matriz if both A and its transpose A* are stochas- 
tic matrices. 
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Proposition 5.2 


Show that a stochastic matrix is doubly stochastic if and only if the sum of the 


entries in each row is 1, i.e. )0,-¢4@j; = 1 for any j € S. 


Proof 


Put A‘ = [b,;]. Then, by the definition of the transposed matrix, b;; = aji. 
Therefore, A’ is a stochastic matrix if and only if 


S- ai = S- bij = l, 
completing the proof. O 


Exercise 5.3 


Show that if P = [p;i]j;,ie5 is a stochastic matrix, then any natural power P” of 
P is a stochastic matrix. Is the corresponding result true for a double stochastic 
matrix? 


Hint Show that if A and B are two stochastic matrices, then so is BA. For the second 
problem, recall that (B.A)‘ = A'B’. 


Exercise 5.4 


Let P= | hs |. Show that 


ar a 2qg-pq-qg 
2p—pqg-p? 1+q*—2q+pq 


Hint This is just simple matrix multiplication. 


We see that there is a problem with finding higher powers of the matrix P. 
When multiplying P? by P, P?, and so on, we obtain more and more compli- 
cated expressions. 


Definition 5.4 


The n-step transition matriz of a Markov chain €, with transition probabilities 
p(j\t), 7,2 € S is the matrix P, with entries 


: /baldlt) = Pl&m = jlo = 4). (5.17) 
y 
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Exercise 5.5 


Find an exact formula for P, for the matrix P from Exercise 5.4. 


Hint Put tn = P(En = O[€o = 0) and yn = P(En = 1/& = 1). Is it correct to suppose 
that pn(0|0) = rn and pn(1|1) = yn? If yes, you may be able use Example 5.1 and 
Exercise 9.1. 


Exercise 5.6 


You may suspect that P, equals P”, the nth power of the matrix P. This holds 
for n = 1. Check if it is true for n = 2. If this is the case, try to prove that 
P, = P” for alln EN. 


Hint Once again, this is an exercise in matrix multiplication. 


The following is a generalization of Exercise 5.6. 


Proposition 5.3 (Chapman—Kolmogorov equation) 


Suppose that €,, n € N, is an S-valued Markov chain with n-step transition 
probabilities p,(j|i). Then for all k,n € N 


Pn+k(Jlt) = >> pn(ils)pe (sli), 2,9 € S. (5.18) 
seS 


Exercise 5.7 
Prove Proposition 5.3. 


Hint pn+k(jli) are the entries of the matrix Prk = pe 


Proof (of Proposition 5.3) 


Let P and P,, be, respectively, the transition probability matrix and the n-step 
transition probability matrix. Since pn(j|i) are the entries of P,, we only need 
to show that P, = P” for all n € N. This can be done by induction. The 
assertion is clearly true for n = 1. Suppose that P, = P”. Then, for 2,7 € S, 
by the total probability formula and the Markov property (5.10) 


Penis = j\€o = 1) 
>> Pléns = Slo = 1, &n = 8)P(En = 8l€o = 4) 


sES 


>> PlEns1 = Jl = 8)P(En = 8|€o = 4) 


rn a 


Pn+1 (j|7) 
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= Y plils)pn(sti) 


sES 


which proves that Pri; = PP,. U 


Exercise 5.8 (random walk) 


Suppose that S = Z. Let n,, n > 1 be a sequence of independent identically 
distributed random variables with P(j, = 1) = pand P(m = —-1) =q=1-p. 
Define £, = )>;.. 7: for n > 1 and & = 0. Show that €, is a Markov chain 
with transition probabilities 


D, if 7=i1+1, 
puUli)=4 a, if gsi, 
0, otherwise. 


En, n > 0, is called a random walk starting at 0. Replacing > = 0 with & =1, 
we get a random walk starting at 7. 


Hint €n41 = £n + M41. Are €) and nn+1 independent? 


Exercise 5.9 


For the random walk €,, defined in Exercise 5.8 prove that 
Pls = slo =i) = (pags pa (5.19) 
2 
ifn+j7—7is an even non-negative integer, and P(E, = j| = 1) = 0 otherwise. 
Hint Use induction. Note that (ns3-2)p equals 0 if |j —7] >n+1. 


Proposition 5.4 


For all p € (0,1) 
P(E, = il& =i) 30, asn— oo. (5.20) 


Proof 
To begin with, we shall consider the case p # 5. When j = i, formula (5.19) 


becomes oe : 
a roe aNtoct Ey? (pq) ’ if n = 2k, 91 
PlEn = to = #) | 0, Pita. Oda: eo 
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Then, denoting a, = tery (pq)*, we have 


Qr41 (2K + 1)(2k + 2) 


= — 4pq < 1. 
ij, 0° (eee Ea 


Hence, a, — 0. Thus, P(f, = i|& = i) > 0. The result follows, since 
P(€ar41 = fo =7) =0-0. 

This argument does not work for p = ‘ because 4pq = 1. In this case we 
shall need the Stirling formula! 


k 
kl ~ VQrk (=) , as k > oo. (5.22) 


Here we use the standard convention: a, ~ 6, whenever - +> lasno ow. 
By (5.22) 


Qn ™ 


V4nk (2) " Oe (oa)! 


nk \ e k 


1 
= —-— 0, as k > ow. 


Vak 


Let us note that the second method works in the first case too. However, in the 
first case there is no need for anything as sophisticated as the Stirling formula. 
D 


Proposition 5.5 


The probability that the random walk €, ever returns to the starting point is 


be | p=q\: 


Proof 


Suppose that & = 0 and denote by fo(n) the probability that the process 
returns to 0 at time n for the first time, i.e. 


fo(n) = P(&, =0,€, 40,1 =1,---,n—1). 


If also po(n) = P (€, = 0) for any n € N, then we can prove that 


(oe) CoO 


>- Po(n) = )> po(n) >> fo(n). (5.23) 
n=! n=0 n 


=1 


‘ See, for example, E.C. Titchmarsh, The Theory of Functions, Oxford University 
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Since all the numbers involved are non-negative, in order to prove (5.23) we 
need only to show that 


po(n) = >— folk)po(n — k) for n> 1. 
k=1 
The total probability formula and the Markov property (5.10) yield 


po(n) =a S| Pl&n = 0,& = 0,6 £40,i=1,---,k-1) 


k=1 
se PHC 01S lom ha) 
k=1 
xP(En = Ol& =0,€ 40,i=1,---,k-1) 


» PE HUG F003 1,45, ba 1) Pe, = lee =0) 
k=1 


I| 


>= fo(k)po(n — k). 
k=1 


Having proved (5.23), we are going to make use of it. First we notice that the 
probability that the process will ever return to 0 equals ae fo(n). Next, from 
(5.23) we infer that 


P(An >1:€& =0) = S© fo(n) 
n=1 
fore) —i foe) ~1 
= 1- (5-0 sa a (> mien) 
n=0 k=0 
Since po(2k) = Haye (pa) and 
“(2k 
S- ( Js! = (1—42)71/?, |z| < 2 (5.24) 
k 4 
k=0 
it follows, that for p#1/2 
P(3n > 1: &, = 0) =1- (1 — 4pq)'/? = 1-|p—q, (5.25) 


since, recalling that gq = 1 — p, we have 1 — 4pq = 1 — 4p + 4p? = (1 — 2p)? = 
(q-p)?. 

The case p = 1/2 is more delicate and we shall not pursue this topic here. 
Let us only remark that the case p = 1/2 needs a special treatment as in 
Proposition 5.4. O 


96 Basic Stochastic Processes 


Exercise 5.10 
Prove formula (5.24). 


Hint Use the Taylor formula to expand the right-hand side of (5.24) into a power 
series. 
Exercise 5.11 (branching process) 


On the island Elschenbieden there lives an almost extinct species called Vugiel. 
Vugiel’s males can produce zero, one, two, --: male offspring with probability 
Po, P1,P2,°°: respectively, where p; > 0 and ae p; = 1. A challenging problem 
would be to find the Vugiel’s chances of survival assuming that each individual 
lives exactly one year. At this moment, we ask you only to rewrite the problem 
in the language of Markov chains. 


Hint The number of descendants of each male has the same distribution. 


Exercise 5.12 


Consider the following two cases: 


1) In Exercise 5.11 suppose that 
N Se 
Pm = ( )pma = a 
m 


for some p € (0,1) and N € N*, where N* = {1,2,3,---}. (Note that 
Pm = 0 if m > N.) Show that 


p= (*")r (1 — p)R3, (5.26) 


Deduce that, in particular, p(j|i) = 0 if 7 > Ni. 


2) Suppose that 


for some A > 0. In other words, assume that each X; has the Poisson 
distribution with mean 4. Show that 
(Ai)? 


pl) = arm ea, (5.27) 


Hint If X, has the binomial distribution P(X; =m) = (")p™(1—p)*-™, mEN, 
then there exists a finite sequence 71,:::,7y of independent identically distributed 
random variables such that P(n? = 1) = ». P(n} = 0) =1—ovand Xi =m! 4+--- + 
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my = m. Hence, X1+--- + Xi = D'_, 0 mM, ice. the sum of Ni independent 
identically distributed candor variables with distribution as above. Hence we infer 


(5.26). 


Proposition 5.6 


The probability of survival in Exercise 5.12, part 2) equals 0 if A < 1, and 1—7* 
if \ > 1, where k is the initial Vugiel population and 7 € (0,1) is a solution to 


r= elt DA, (5.28) 


Proof 


We denote by $(2), 1 € N the probability of dying out subject to the condition 
&) =i. Hence, if A = {€, =0 for n € N}, then 


o(i) = P (Algo = 8). (5.29) 


Obviously, ¢(0) = 1 and the total probability formula together with the Markov 
property (5.10) imply that for each i € N 


Ms: 


(i) PA == 9) PG = 9 =4) 
j=0 
=> P(A =) PG =jle =) 
47=0 
= > o(3)p(l8). 
j=0 


Therefore, the sequence (i), 2 € N is bounded (by 1 from above and by 0 from 
below) and satisfies the following system of equations 


> $(i)PGlE): iEN, (5.30) 
o(0) = 1. 


So far, we have not used any particular distribution of X;. From now on, we 
shall assume that the X; have the Poisson distribution. Hence, by Exercise 5.12, 
p(jlt) = (ay en, It is not an easy problem to find a solution to (5.30), even in 
this special case. ¢(z) is the probability that the population will die out, subject 
to the condition that initially there were 2 individuals. Since we assume that 
reproduction of different individuals is independent, it is reasonable to make 


the following Ansatz: 


ass 

— 
~. 

A 
I| 


Ali\ = AIAQYY. GEN. (5.31) 
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for some A > OQ. Although it is possible to prove this Ansatz, we shall not 
do so here. Note that the boundary condition ¢(0) = 1 implies that A = 1. 
Substituting (5.31) (with A = 1 and r := ¢(1)) into (5.30), we get 


CO ‘ . 
(tA) a 
tt J 1 
r= yr, 

j=0 


a | 
ew ik > —(ird)? = ee Aeirr. 
j=0 7 


Hence, r should satisfy 
r= el DA, (5.32) 


Since the function g(r) = e'"~))4, r € [0,1], is convex, there exist at most 
two solutions to the equation (5.32). Obviously, one of them is r = 1. A bit of 
analysis, not included here, shows the following: 


1) If \ < 1, then the only solution to (5.32) in [0,1] is r = 1. 


2) If A > 1, then there exists a second solution 7 € (0,1) of the equation 
(5.32). . 


In case 1) the situation is simple. We have ¢(i) = 1 for all 7, and thus the 
probability of extinction is 1 for any initial number of individuals. Case 2) is 
slightly more involved. The first question we need to address is which of the 
two solutions of (5.32) gives the correct value of ¢(1)? Recall that py = ae 
Define 


[o @) (oe) k 
F(z) = Spex" = > se Nak =eVe™. ae) <1. (5.33) 
k=0 k=0 — 


Since P(E, = O|& = 1) = Po and 


P (€ =0|f =1) = So P(€: = 0]; =i) P(& =i = 1) 


i=—0 
(oe) 


S| (po)*pi = F (po) = F(F(0)), 


i=0 


we guess that the following holds: 
P(n = O|fo = 1) = F)(0), (5.34) 


where F”) is the n-fold composition of F. To prove (5.34) it is enough to prove 
it for n, while assuming it holds for n — 1. We have 


P (€, = 0|6 =1) = S>P(& =O =i) P(& = il =1) 
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= S> iP (En—1 = O|& = 2) 


1=0 


> pi (P(En-1 = Ol = 1)" 


1=0 


yp [re—)(0)] = F(FI"7)(0)) = F™ (0). 
1=0 


Since the event {€, = 0} is contained in events {€,4, = 0} for all nm € N, we 
have 


P {€, = 0, for some n € Nf = 1} 
= im, P {Em = Olé = 1} 


¢(1) 


by the Lebesgue monotone convergence theorem. Therefore, we infer that 


o(1) = lim F'™ (0). 


noo 
With F(z) = x we only need to show that 
F'%(0) <#, neNn. (5.35) 


Indeed, once the inequality (5.35) is proven, we infer that ¢(1) < 7 and thus 
(1) = 7. We shall prove (5.35) by induction. It is obviously valid for n = 0, so 
we need to study the inductive step. We have 


FO(0) = F(FO-N(0)) < FA) =F, 


since F is increasing. We conclude that in the case \ > 1 the population will 
become extinct with positive probability. 
In the simplest example of the binomial distribution case, i.e. when N = 1, 


equations (5.30) become 
ee ae pee og 
oi) = 5 60)()r'a-ny, ie 
j=0 ¢ 
Since ¢(0) = 1, (1) satisfies 


o(1) =q+9(1)p 


with gq = 1 — p. Hence, trivially, 6(1) = 1. Then, by induction, one proves that 
g(t) = 1. Therefore, whatever the initial number of individuals, extinction of 
the species is certain. U 
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Remark 5.2 


The method presented in the last solution works for any distribution of the 
variables X;. It turns out that the mean value A of X, plays the same role as 
above. One can show that if A < 1, then the population will become extinct 
with probability 1, while for A > 1 the probability of extinction is larger than 
0 and smaller than 1. 


Exercise 5.13 


On the, now familiar, island of Elschenbieden the question of survival of the 
Vugiel is a hot political issue. The (human) population of the island is N. Those 
who believe that action should be taken in order to help the animals preach 
their conviction quite convincingly. For if a supporter discusses the issue with a 
non-supporter, the latter will change his mind with probability one. However, 
they do so only in face-to-face encounters. Suppose that the probability of an 
encounter of exactly one pair of humans during one day is p and that with 
probability qg this pair is a supporter—non-supporter one. Write down a Markov 
chain model of this situation. Neglect the probability of two or more encounters 
during one day. 


Hint On each day the number of supporters can either increase by 1 or remain un- 
changed. What is the probability of the former? 


Exercise 5.14 (queuing model) 


A car wash machine can serve at most one customer at a time. With probability 
p, 0 < p< 1, the machine can finish serving a customer in a unit time. If this 
happens, the next waiting car (if any) can be served at the beginning of the 
next unit of time. During the time interval between the nth and (n + 1)st unit 
of time the number of cars arriving has the Poisson distribution with parameter 
A > 0. Let €, denote the number of cars being served or waiting to be served 
at the beginning of unit n. Show that €,, n € N, is a Markov chain and find 
its transition probabilities. 


Hint Let Z,, n = 0,1,2,--- be a sequence of independent identically distributed 
random variables, each having the Poisson distribution with parameter A. Then €,41— 
En — Zn equals —1 or 0. 


Remark 5.3 


In the last model we are interested in the behaviour of €, for large values of n. 
In particular, it is interesting to determine whether the limit of €, or that of Eg, 
(as n > oo) exists. In Exercise 5.36 we shall find canditinns which onarantes 
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the existence of a unique invariant measure and imply that the Markov chain 
in question is ergodic. 


5.2 Classification of States 


In what follows we fix an S-valued Markov chain with transition matrix P = 
[p(j|t)]j,ies, where S is a non-empty and at most countable set. 


Definition 5.5 


A state 7 is called recurrent if the process €, will eventually return to 7 given 
that it starts at 1, i.e. 


P(€, = 7% for some n > 1| =1) = 1. (5.36) 


If the condition (5.36) is not satisfied, then the state z is called transient. 


Theorem 5.1 


Show that for a random walk on Z with parameter p € (0,1), the state 0 is 
recurrent if and only if p = 1/2. Show that the same holds if 0 is replaced by 
any other state i € Z. 


Proof 


We know from (5.25) that P(€, = i for some n > 1| = 7) = 1 — |p — q| for 
anyz€Z.OU 


Definition 5.6 


We say that a state i communicates with a state j if with positive probability 
the chain will visit the state 7 having started at 2, ie. 


P(E, = j for some n > Of = 7) > 0. (5.37) 


If 1 communicates with j, then we shall write 1 — 3. We say that the state 2 
intercommunicates with a state 7, and write i 4 j, ifi 47 andj > 1. 


Exercise 5.15 


Show that i > 7 if and only if p,(j|t) > 0 for some k > 1. 
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Hint Recall that pz (jli) = P(€ = j|€o = 1). 


Exercise 5.16 

Show that 

l) iid, 

2) ifi #7 then 7 4 7 and 

3) ficogjoktheniok. 


In other words, show that © is an equivalence relation on S. 


Hint 1) and 2) are obvious. For 3) use the Chapman—Kolmogorov equations. 


Exercise 5.17 
For |z| < 1 and j,i € S define 


Pyi(z) = S— pnljlé)2”, (5.38) 
n=0 
FA) = OU (5.39) 


where fn(7|2) = P(En = ; Ee #9,k =1,---,n—1]& = 7). Show that the power 
series in (5.38)—(5.39) are absolutely convergent for |z| < 1 and that 
PA) = Lye) Pj (2), i 7 4, (5.40) 
P,;(x) 1+ Fy (x) Py (a). (5.41) 


II 


Hint Note that |pn(j|t)| <1, so the radius of convergence of the power series (5.38) 
is > 1. 


Exercise 5.18 
Show that limg ~1 P3;(2) = OP. 9 Pn(Jly) and limg ~1 Fyj(2) = oP fa (JIJ)- 


Hint Apply Abel’s lemma’: If a; > 0 for all k > 0 and limsup,_,,, */|ax| <1, then 
lime 71 oP 9 apz* = > po 2k, nO matter whether this sum is finite or infinite. 


* For example, see: W. Rudin, Principles of Mathematical Analysis, McGraw-Hill 
Book Company, New York 1976. 
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Exercise 5.19 


Show that a state j is recurrent if and only if }°, pr(jl|j) = co. Deduce that 
the state 7 is transient if and only if 


>= Pnlild) < 00. (5.42) 


Show that if j is transient, then for eachi € S 


Sn) < 00. (5.43) 


Hint If j is recurrent, then Fj;(z) 4 0) fa(jlj) = 1 as x (7 1. Use (5.41) in 
conjunction with Abel’s Lemma. 


Exercise 5.20 


Kop: | show that 


p Leg 


For a Markov chain €, with transition matrix P = | 


both states are recurrent. 


Hint Use Exercise 5.19 and 5.5. 


One may suspect that if the state space S is finite, then there must exist 
at least one recurrent state. For otherwise, if all states were transient and 
S = {1,2,---,N}, then with positive probability a chain starting from 1 would 
visit 1 only a finite number of times. Thus, after visiting that state for the 
last time, the chain would move to a different state, say iz, in which it would 
stay for a finite time only with positive probability. Thus, in finite time, with 
positive probability, the chain will never return to states 1 and 22. By induction, 
in finite time, with positive probability, the chain will never return to any of 
the states. This is impossible. The following exercise will give precision to this 
argument. : 


Exercise 5.2] 


Show that if €, is a Markov chain with finite state space S, then there exists 
at, least one recurrent state i € S. 


Hint Argue by contradiction and use (5.43). 


The following result is quoted here for reference. The proof is surprisingly 
difficult and falls beyond the scope of this book. 
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Theorem 5.2 


A state 7 € S is recurrent if and only if 
P(€, =) for infinitely many n|& = 7) = 1, 
and is transient if and only if 


P(€, = j for infinitely many n|& = 7) = 0. 


Definition 5.7 


For an S-valued Markov chain £,,n € N, a state i € S is called null-recurrent 
if it is recurrent and its mean recurrence time m, defined by 


(oe) 


mi:= >> nfa(lili) (5.44) 


n=0 


equals oo. A state 2 € S is called positive-recurrent if it is recurrent and its 
mean recurrence time m; is finite. 


Remark 5.4 


One can show that a recurrent state 7 is null-recurrent if and only if p,(i|i) — 0. 


We already know that for a random walk on Z the state 0 is recurrent if and 
only if p = 1/2, i.e. if and only if the random walk is symmetric. In the following 
problem we shall try to answer if 0 is a null-recurrent or positive-recurrent state 
(when p = 1/2). 

Exercise 5.22 


Consider a symmetric random walk on Z. Show that 0 is a null-recurrent state. 
Can you deduce whether other states are positive-recurrent or null-recurrent? 


Hint State 0 is null-recurrent if and only if ye nfn(0|0) = co. As in Exercise 5.18, 
>on Mfn(0/0) = lim, 1 Foo(x), where Foo is defined by (5.39). : 


Exercise 5.23 


For the Markov chain €, from Exercise 5.20 show that not only are all states 
recurrent, but they are positive-recurrent. 


Hint Calculate fn(0|0) directly. 
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The last two exercises suggest that the type of a state 2 € S, i.e. whether 
it is transient, null-recurrent or positive-recurrent is invariant under the equiv- 
alence ++. We shall investigate this question in more detail below, but even 
before doing so we need one more notion: that of a periodic state. 


Definition 5.8 


Suppose that €,, n € N, is a Markov chain on a state space S. Let 1 € S. We 
say that 2 is a periodic state if and only if the greatest common divisor (gcd) 
of all n € N*, where N* = {1,2,3,---}, such that pp(i|t) > O is > 2. Otherwise, 
the state 7 is called aperiodic. In both cases, the gcd is denoted by d(7) and is 
called the period of the state 7. Thus, 2 is periodic if and only if d(z) > 2. A 
state 2 which is positive recurrent and aperiodic is called ergodic. 


Exercise 5.24 


Is this claim that pq(;)(t|t) > 0 true or not? 


Hint Think of a Markov chain in which it is possible to return to the starting point 
by two different routes. One route with four steps, the other one with six steps. 


One of the by-products of the following exercise is another example of the 
type asked for in Exercise 5.24. 


Exercise 5.25 


Consider a Markov chain on S = {1,2} with transition probability matrix 


PS | , Ve | . This chain can also be described by the graph in Figure 5.1. 


Find d(1) and d(2). 


1/2 


Figure 5.1. Transition probabilities of the 


1/2 Markov chain in Exercise 5.25 


Hint Calculate P? and P*. This can be done in two different ways: either algebraically 
or, probabilistically. 
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Proposition 5.7 

Suppose that 7,7 € S and i © 7. Show that 

1) 7 is transient if and only if 7 is; 

2) 2 is recurrent if and only if 7 is; 

3) zis null-recurrent if and only if 7 is; 

4) 1 is positive-recurrent if and only if 7 is; 

5) 7 is periodic if and only if 7 is, in which case d(i) = d(j); 


6) 2 is ergodic if and only if 7 is. 


Proof 


It is enough to show properties 1), 4) and 5). Since i + j one can find n,m € N 
such that pm(j|t) > 0 and pr(ilj) > 0. Hence € := pm(jlt)pn(i|7) is positive. 
Let us take k € N. Then by the Chapman—Kolmogorov equations 


Pm+ktn(jl9) = D> Pm(i|s)pe(sir)Pn(rlj) > Pm (Ilé)pe(ilé)Pn (i[7) = eve (ili). 
r,sES 


By symmetry 


Pn+k+m(ili) = Y— Palils)pe(slr)Pm(rlé) > Palils)pe(I|F)Pm (lt) = epe (ld). 
r,sES 


Hence, the series >), pxe(t|t) and )>, px(j|7) are simultaneously convergent or 
divergent. Hence 1) follows in view of Exercise 5.19. 
To prove 5) it is enough to show that 


d(i) < d(9). 
Using the first inequality derived above, we have 
Pntk+m(t|t) > epe(J|9) 
for all k € N. From this inequality we can draw two conclusions: 
(a) d(i)|n + m, since by taking k = 0 we get prim(ili) > 0; 
(b) if pe(gl7) > 0, then pryegm (tli) > 0. 


From (a) and (b) we can see that d(i)|k provided that p,(j|j) > 0. This proves 
what is required. O 
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Exercise 5.26 


Show that the following modification of 2) above is true: If 7 is recurrent and 
1 — 7, then 7 + 2. Deduce that if 7 is recurrent and 7 > j, then 7 is recurrent 
and 7 #1. 


Hint Is it possible for a chain starting from 7 to visit 7 and then never return to 2? 
Is such a situation possible when 7 is a recurrent state? 


The following result describes how the state space S can be partitioned into 
a countable sum of classes. One of these classes consists of all transient states. 
Each of the other class consists of interconnecting recurrent states. If the chain 
enters one of the classes of second type, it will never leave it. However, if the 
chain enters the class of transient states, it will eventually leave it (and so never 
return to it). We begin with a definition. 


Definition 5.9 


Suppose that £,, n € N, is a Markov chain on a countable state space S. 


1) A set C C S is called closed if once the chain enters C it will never leave 
it, i.e. 
P (& € S\C for some k > n|€, € C) = 0. (5.45) 


2) Aset CC S is called irreducible if any two elements i,j of C intercommu- 
nicate, i.e. for all 7,7 € C' there exists an n € N such that p,{j|i) > 0. 


Theorem 5.3 
Suppose that £,, n € N, is a Markov chain on a countable state space S. Then 
N 
S=TU[JCj, (disjoint sum), (5.46) 
j=l 


where T' is the set of all transient states in S and each C; is a closed irreducible 
set of recurrent states. 


Exercise 5.27 


Suppose that €,, n € N, is a Markov chain on a countable state space S. Show 
that a set CC S is closed if and only if p(j|i) = 0 for allt € CandjeS\C. 


Hint One implication is trivial. For the other one use the countable additivity of the 
measure P. 
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Proof (of Theorem 5.3) 


Let R = S\T denote the set of all recurrent states. If i 4 37, then both i 
and j belong either to JT or to R. It follows that the interconnection relation 
+> restricted to R is an equivalence relation as well. Therefore, R = Cie oF 
C; = [s;], 5; € &. Here N denotes the number of different equivalent classes. 
Since by definition each C; is an irreducible set, we only need to show that it 
is closed. But this follows from Exercise 5.26. Indeed, if 1 € C, and i > 3, then 
14> j, andsojeée Cy. O 


5.3 Long-Time Behaviour of Markov Chains: 
General Case 


For convenience we shall denote the countable state space S by {1,2,3,---} 
when S$ is an infinite set and by {1,2,---,n} when S is finite. 


Proposition 5.8 


Let P = [p(j|z)] be the transition matrix of a Markov chain with state space S. 
Suppose that for all 1,7 € S 


tim pn (34) =: (5.47) 
(In particular, the limit is independent of 2.) Then 
1) = mj <4, 


2) ee PGlt)ms = 75; 
3) either }/,7j = 1, or 7; = 0 for all j € S. 


Proof 


To begin with, let us assume that S is finite with m elements. Using the 
Chapman-Kolmogorov equations (5.18), we have 


m m 
om = Dom = DS lim pall) 
j=l g=1 


jES 


lI 


om 

li efe — e = 

ie 2 Pel ae 
J= 
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since 2 Pn(j|t) = 1 for any n € N (see Exercise 5.2). This proves 1) and 3) 
simultaneously. Moreover, it shows that the second alternative in 3) can never 
occur. To prove 2) we argue in a similar way. Let us fix 7 € S and (an auxiliary) 
k € S. Then, 


2 Pili = 2, itn, PCJT)Pm le) 


I! 


dim > p(ili)Pn (ilk) = lim, Pras Gk) = my, 
i=l 


since )-\", p(j|t)pn(t|K) = pn4i(j|k) by the Chapman-Kolmogorov equations. 

When the set S is infinite, we cannot just repeat the above argument. The 
reason is quite simple: in general the two operations lim and )> cannot be 
interchanged. They can when the sum is finite, and we used this fact above. 
But if S 7s infinite, then the situation is more subtle. One possible solution of 
the difficulty is contained in the following version of the Fatou lemma. 


Lemma 5.1 (Fatou) 
Suppose that a;(n) > 0 for j,n € N. Then 


S¢ lim inf a;(n) < lim inf y a;(n). (5.48) 
a mG 
Moreover, if a;(n) < ; for j,n € N and 97, b; < oo, then 


lim sup )_a;(n) < Slim sup a;(7n). (5.49) 


J 


Using the fact that for a convergent sequence lim and lim inf coincide, by 
the Fatou lemma we have 


(oe) CO 
27) = 27) = 2 dim, poll 


(oe) 
Himint D Pall = lim inf 1 =] 
I= 


1A 


since, as before, i Pn(j|t) = 1 for any n € N. This proves 1). A similar 
argument shows 2). Indeed, with 7 € S and k € S fixed, by the Chapman- 
Kolmogorov equations and the Fatou lemma we have 


do pclim: =D) lim plsli)pn (dl) 
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< lim inf 2, PU) Pn(ele) = lim inf pn4i (j/k) = 73. 


To complete the proof of 2) suppose that for some k € S 


S— pl ki) < Tr. 
i=1 


Then, since )) ;¢57j = )1j44 7 +7, by the part of 2) already proven we have 


se => (So in) + Dap (kli) 7 


j=l j#k t 
CO 600 
= 2 Ll (j|2) 1)m; = > Da (j|t)a 
j=l =1 j 
00 ore) 00 
= Sn (Sri) = Se 
i=] j i=1 


We used the fact that as p(j|i) = 1 together with (5.65). This contradiction 
proves 2). 
In order to verify 3) observe that by iterating 2) we obtain 


S— pal Jlé)m = 1. 


Hence, 


ee 
| 


a fee le aes 
Jim. 2, Palgliyms 
t 
> lim, Pa (Jlé)m ic =; dt. 
r 1 t 


Therefore, the product 7; (}_,; 7 — 1) is equal to 0 for all j € S. As a result, 3) 
follows. Indeed, if 50,7; #1, then 7; = 0 for ally Ee S.O0 


Definition 5.10 


A probability measure p := >> jes #9; 18 an invariant measure of a Markov 
chain €,, n € N, with transition probability matrix P = [p(j|t)] if for alln E N 
and allj ES 


S5 pals li) ei = Hy. 


iES 
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Exercise 5.28 


Under the assumptions of Proposition 5.8 show that if >> jes = 1, then 
16 := Yo eg 75; is the unique invariant measure of our Markov chain. Here 6; 
is the Dirac delta measure at 7. 


Hint ps is an invariant measure of a Markov chain £n, n € N, if and only if for each 
n EN, the distribution of €, equals 1, provided the same holds for €o. 
Exercise 5.29 


Show that if 7; = 0 for all 7 € S, then there is no invariant measure. 


Hint Look closely at the uniqueness part of the solution to Exercise 5.28. 


The following exercise shows that a unique invariant measure may exist, 
even though the condition (5.47) is not satisfied. 


Exercise 5.30 


Find all invariant measures for a Markov chain whose graph is given in Fig- 
ure 5.2. 


l Figure 5.2. Transition probabilities of the 
Markov chain in Exercise 5.30 


Hint Find the transition probability matrix P and solve the vector equation Px = 7 
for 7 = (7, 72), subject to the condition m + 72 = 1. 


We shall study some general properties of invariant measures. Above we 
have seen examples of Markov chains with a unique invariant measure. In what 
follows we shall investigate the structure of the set of all invariant measures. 


Exercise 5.31 


Show that if 4 and v are invariant measures and 6 € (0, 1], then (1 — 6)y + Ov 
is also an invariant measure. 


mr: ‘ A 1 i Bee > Bra oe A ee ee ee ee ee 
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Exercise 5.32 

Show that if uw is an invariant measure of a Markov chain €,, n € N with state 
space S, then supp zp C S\T, where T denotes (as usual) the set of all transient 
states. 


Hint If j is a transient state, then p,(j|z) > 0 for alli € S. 


The above result shows that there is a close relationship between invariant 
measures and recurrent states. Below we shall present without proof a couple 
of results on the existence of such measures and their properties. 


Theorem 5.4 


Suppose that €,, n € N, is a Markov chain on a state space S = T UC, where 
T is the set of all transient states and C’ is a closed irreducible set of recurrent 
states.> Then there exists an invariant measure if and only if each element of C 
is positive-recurrent. Moreover, if this is the case, then the invariant measure 
is unique and it is given by uw = )>, wid;, where 

i 

hi = — 
mi 


with m; being the mean recurrence time of the state i, see (5.44). 


Note, that by Exercise 5.32, the unique invariant measure in Theorem 5.4 
is supported by C. 


Remark 5.5 


it-C. = ee C’;, where each Cj is a closed irreducible set of recurrent states, 
then the above result holds, except for the uniqueness part. In fact, if each 
element of some Cj is positive-recurrent, then there exists a invariant measure 
jt; Supported by C;. Moreover, yz; is the unique invariant measure with support 
in C;. In the special case when each element of C' is positive-recurrent, every 
invariant measure p is a convex combination of the invariant measures p;, 


j€ {1,---, NF}. 


Theorem 5.5 


Suppose that €,,  € N, is a Markov chain with state space S. Let 7 € S bea 
recurrent state. 


3 Hence in the decomposition (5.46) the number N of different classes of recurrent 


minh we Sa Reale ee I 


saosin geensnmntermmermimeragmteet TMT PY RTT RETA 
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1) If 7 is aperiodic, then 


ia ] 
Pr(jlj) + —. (5.50) 
m5 
Moreover, for any 7 € S, 
bi Pyi(1 
pa (ili) + 72 (5.51) 
m 


j 
where F;;(1) is the probability that the chain will ever visit state 7 if it 
starts at 7, see (5.39), and where m; is the mean recurrence time of state 7, 
see (5.44); 


2) If 7 is a periodic state of period d > 2, then 


d 
nd(7|7 —. 5.52 
Pna(j|j) > 7 (5.52) 


Exercise 5.33 


Suppose that €,, n € N, is a Markov chain with state space S. Let 7 € S bea 
transient state. Show that for any i € S 


Pn(j|t) > 0. (5.53) 


Hint Use Exercise 5.19. 


Definition 5.11 


A Markov chain £,,n € N, with state space S is called ergodic if each i € S is 
ergodic, i.e. each state z € S is positive, recurrent and aperiodic. 


Exercise 5.34 


Show that if €,, n € N, is an ergodic irreducible Markov chain with state 
space S, then p,(j|i) + 1; as n > oo for any j,i € S, where m = )°, 176; is 
the unique invariant measure. 


Hint Use Theorem 5.5. You may assume as a known fact that if 7 is recurrent and 
ij, then F;;(1) = 1. 


Exercise 5.35 


Use the last result to investigate whether the random walk on Z has an invariant 
measure. 


Hint Thea Rvarcica 4 QO 
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Below we shall see that a converse result to Theorem 5.5 is also true. 


Theorem 5.6 


Suppose that €,, n € N, is an irreducible aperiodic Markov chain with state 
space S. Then €,, n € N, is ergodic if and only if it has a unique invariant 
measure. 


Proof 


The ‘if’ part is proved in Exercise 5.34. We shall deal with the ‘only if’ part. 
Suppose that 7 = > ; 730; is the unique invariant measure of the chain. Then 
m; > 0 for some j € S. Recall that due to Theorem 5.5 and the Exercise 5.33, 
limyn+4o0 Pn(jlt) exists for all i,7 € S. 

Since >0; Pn(jlt)m; = 7;, by the Fatou lemma (inequality (5.49)) 


>_ Jim, palili)m 2 limsup ) | Pn(Jli) mi = yj. 
a 2 


Hence, there exists an i € S such that limp. pn(j|i)t; > O. Therefore 
limpsoo Pn(jlt) > 0, which in view of Theorem 5.5 implies that m; < ov. 
Thus, 7 is an ergodic state and, since the chain is irreducible, all states are 
ergodic as well. 0 


Exercise 5.36 


Prove that if there exists an invariant measure for the Markov chain in Exer- 
cise 5.14, then A’ := ae, jq; < 1. Assuming that the converse is also true, 
conclude that the chain is ergodic if and only if \’ < 1. Show that if such an 
invariant measure exists, then it is unique. 


Hint Suppose that 7 = eer 7j6; iS an invariant measure. Write down an infinite 
system of linear equations for 7;. If you don’t know how to follow, look at the solution. 


5.4 Long-Time Behaviour of Markov Chains 
with Finite State Space 


As we have seen above, the existence of the 1; plays a very important role 
in the study of invariant measures. In what follows we shall investigate this 
question in the case when the state space S is finite. 
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Theorem 5.7 


Suppose that S is finite and the transition matrix P = [p(j|t)] of a Markov 
chain on S satisfies the condition 


dng € N de > 0: pn, (jlt) > €, 1,9 ES. (5.54) 


Then, the following limit exists for all 7,7 € S and is independent of :: 


lim pr(jlt) = 75. (5.55) 
m— OO 
The numbers 7; satisfy 
1 >0,j€S and Son; =1. (5.56) 
jes 


Conversely, if a sequence of numbers 7;, 7 € S satisfies conditions (5.55)—(5.56), 
then assumption (5.54) is also satisfied. 


Proof 


Denote the matrix P™° = [pp, (j|t)] by Q@ = [q(j|z)]. Then the process 7, = £kno; 
k €N, is a Markov chain on S with transition probability matrix Q satisfying 
(5.54) with ng equal to 1. Note that pen, (j|t) = ge (j|t) due to the Chapman— 
Kolmogorov equations. Suppose that the properties (5.55)—(5.56) hold true 
for Q. In particular, limgoo Pino (J|t) = 7; exists and is independent of 1. We 
claim that they are also true for the original matrix P. Obviously, one only 
needs to check condition (5.55). The Chapman—Kolmogorov equations (and 
the fact that S is finite) imply that for any r = 1,---,ng —1 


S— Pn (j|s)p,(s|t) aa S- 1jPr (s|z) 


sES sES 


Tj S- pr (sli) =e. 


sES 


Pknot+r Gj 2) 


Therefore, by a simple result in calculus, according to which, if for a sequence 
an, n € N, there exists a natural number ng such that for each r € {0,1,---,no- 
1} the limit limz_,.6 Gkno+r exists and is r-independent, then the sequence ay 
is convergent to the common limit of those subsequences, we infer that (5.55) 
is satisfied. 

In what follows we shall assume that (5.54) holds with no = 1. Let us put 
Po(j|z) = 6;, and forge S$ 


Mn(j) = min Pr (312) 
M,(j) := max Pn(j|*) 
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Observe that Mo(j) = 1 and mo(j) = 0 for all 7 € S. From the Chapman-— 
Kolmogorov equations it follows that the sequence M,,(j), n € N, is decreasing, 
while the sequence m,(j), n € N, is increasing. Indeed, since >, p(k|t) = 1, 


keS 

min pa(j|k) >, p(Ali) 
keS 

min pn(jlk) = mn(3): 


Pn+1 (3 |?) 


IV 


\| 


Hence, by taking the minimum over all 7 € S, we arrive at 
Mn+1(J) a min Pr+41 (9 |?) 2 Mn(J). 


Similarly, 


Pnti(jlt) = S— pn(sle)p(klt) 


keS 

max Pn(jlk) >) P(Klt) 
keES 

max Pn(j|k) = M,(j).- 


lA 


II 


Hence, by taking the maximum over all i € S, we obtain 
Mn+i(j) = max Pa+1 (9/2) < M,(j). 


Since M,,(j) > mn(j), the sequences M,,(7) and m,(j) are bounded from below 
and from above, respectively. As a consequence, they both have limits. To show 
that the limits coincide we shall prove that 


tim (Ma(j) —mn(3)) = 0. (5.57) 
For n > 0 we have 
Pn+ildli) = )~ pn(Jls)p(slé) (5.58) 
séS 
= S~ pn(Jls) [p(slt) — epn(sl3)] + € — Pn(jls)Pn(sl3) 
secs sES 


>| prs) [p(slé) — epn(sls)] + pen (dls) 
scS 


by the Chapman—Kolmogorov equations. The expression in square brackets is 
> 0. Indeed, by assumption (5.54), p(s|z) > € and p,(s|j) < 1. Therefore, 


SPS Ere TT ce reMTUNNE EEE 
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Pnsilili) 2 mig Pa(ils) > [r(slt) ~ emn(slj)] + epan(al) (5.59) 


= (1- E)m,(J) + €prn(JlJ). 
By taking the minimum over 7 € S, we arrive at 
mn+1(j) > (1 —€)malj) + epan(ils). (5.60) 


Recycling the above argument, we obtain a similar inequality for the sequence 


Mnsi(j) < (1 — €)Mn(Z) + Epon (Ila). (5.61) 
Thus, by subtracting (5.60) from (5.61) we get 
Mn+i(9) — Mngi(9) < (1 — €) (Mn(Z) — MnlJ))- (5.62) 


Hence, by induction 
M,(j) — mMn(9) < A-€)", nen 


This proves (5.57). Denote by 7; the common limit of M,(j) and m,(j). Then 
(5.55) follows from (5.57). Indeed, if i, 7 € S, then 


To prove that 7; > 0 let us recall that m,(j) is an increasing sequence and 
m(j) > € by (5.54). We infer that 7; >e. 0 
Exercise 5.37 


Show that p,(j|t) + 7; at an exponential rate. 


Hint Recall that mn(j) <7; < Mn(j) and use (5.62). 


The above proves the following important result. 


Theorem 5.8 


Suppose that the transition matrix P = [p(j|z)] of a Markov chain €,, n € N, 
satisfies assumption (5.54). Show that there exists a unique invariant measure 
pu. Moreover, for some A > 0, and a < 1 


lPn(J|t) —7;| < Aa", i,7 ES nEN (5.63) 
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Proof (of the converse part of Theorem 5.7) 


Put 1 
—€ = —min7,;. 
24 8 


Since pr(j|t) > 7; for all 1,7 € S, there is an ng € N, such that px (j|t) > € for 
all for k > no and (1,7) € S*. Putting k = ng proves that (5.54) is satisfied. 

Let us observe that we have used only two facts: 7; > 0 for all 7 € S, and 
Pn(jlt) > 7; for alli,7 ES. 0 


Exercise 5.38 


Investigate the existence and uniqueness of an invariant measure for the Markov 
chain in Proposition 5.1. 


Hint Are the assumptions of Theorem 5.7 satisfied? 


Remark 5.6 


The solution to Exercise 5.38 allows us to find the unique invariant measure by 
direct methods, i.e. by solving the linear equations (5.79)—(5.80). 


Exercise 5.39 ~ 


Find the invariant measure from Exercise 5.38 by calculating the limits (5.55). 


Hint Refer to Solution 5.5. 


Exercise 5.40 


Ian plays a fair game of dice. After the nth roll of a die he writes down the 
maximum outcome €, obtained so far. Show that €, is a Markov chain and find 


its transition probabilities. 
Hint €n41 = max{én, Xn+1}, where X, is the outcome of the kth roll. 


Exercise 5.41 


Analyse the Markov chain described in Exercise 5.40, but with fair die replaced 


by a fair pyramid. 


Hint A pyramid has four faces only. 


£ 
: 
f 
E 
t 
E 
i 
5 
i 
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Exercise 5.42 


Suppose that a > 1 is a natural number. Consider a random walk on S = 
{0,1,---,a} with absorbing barriers at 0 and a, and with probability p of 
moving to the right and probability g = 1 — p of moving to the left from any 
of the states 1,...,a@ — 1. Hence, our random walk is a Markov chain with 
transition probabilities 


if l<i<a-1,j=i41, 
if 1<i<a—-1,j=i-1, 
i te 7 = ora = 7 =a, 
otherwise. 


p(j|t) = 


Or? 'S 


Find, (a) all invariant measures (there may be just one), (b) the probability of 
hitting the right-hand barrier prior to hitting the left-hand one. 


Hint For (a) recall Exercise 5.30 and for (b) Exercise 5.12. 


5.5 Solutions 


Solution 5.1 : 


First we give a direct solution. With A, and B, being the events that the phone 
is free or busy in the nth minute, we have y, = P(B,). The total probability 
formula then yields, for n € N, 


P(Bn41) = P(Bn4i|An)P(An) + P(Bn4i|Bn)P(Bn) 


Ynt1 =P + (1—p-q)yn. (5.64) 
As before, assuming for the time being that y = lim, yn exists, we find that 


a = =a ‘ =). Pp... F _ o _ 
y=p+(l-p q)y, and so y = 54. In particular, -P- =p+ (1 p ash. 


Subtracting the last equality from (5.64), we see that {yn — ah is a geometric 
sequence, so that yn — a = (yo — B-) (1—p-—q)". Since yp = 1, some simple 
algebra leads to the formula 

Pp q 
=P 4-1 —p—q)", nen 
p+q ptq 


The last formula can be used to prove that the lim, yn exists and is equal to 


Yn 


= 
p+q 
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Another approach is to use the results of Example 5.1. Since zo = 1 — Yo, 
by (5.8) we have 


Yn |= ay = 1-4 — (1-w - 2) pa" 


pt+q pt+q 
era ee ae 
which agrees with the first method of solution. 
Solution 5.2 
We have to show that, Dyes PUI?) = 1 for every 1 € S. We have 


Y= pili) = 45 P(& = 5lé = 4) 


jes jes 
= P(Ujes {1 = J} |& = 7) = P(& € Sl =i) 
a ee 


Solution 5.3 


Suppose that A = [aj;],,ies and B = [b;:];,1e5 are two stochastic matrices. If 
C = BA, then cj; = 5°, bjnaxi. Hence, for any i € S 


cS. (= bas =S> | do bjani 
j i \k R\G 
= a Sys ani = Dan; = 1, 
kG k 


where the last two equalities hold because B and A are stochastic matrices. 
We have used the well-known fact that 


yo a; = a 5.65) 
ij ji 
for any non-negative double sequence (a;;)?5_, (see, for example, Rudin’s book 
cited in the hint to Exercise 5.18). The above argument implies that P? is a 
stochastic matrix. The desired result follows by induction. 
To prove that P” is a double stochastic matrix whenever P is, it is enough 
to observe that AB is a double stochastic matrix if A and B are. The latter 
follows because (AB)! = BtAt. 
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Solution 5.4 


Some simple algebra gives 


p? _ pee ae i a 
P(l-p)+(1-a@)p pqat+(1-qg)(1-@) 
a i 2q — pq — | 
2p-pa-p? 1+q°—-2q+p¢9 
Solution 5.5 


Put zn = P(E, = O]€> = 0) and yn = P(E, = 1/& = 1). We have calculated 
the formulae for z, and y, in Example 5.1 and Exercise 5.1. Since also 


p p 
1-2, = —— —- ——(1-p-4q)", 
"= o4p a+p 
q q 
L-gn = 1 -~—L(1-p-9)", 
"  q+p ep 


we arrive at the following formula for the n-step transition matrix: 


geo + qep(l—P— 9)" ap gap (lL -P- 9)” 
pap ae) Gee ee ee Pg) 


Solution 5.6 


Simplifying, we have 


ep aR O)- Sie apg): 
pot Ge bg)? Maa lp ag) 


Py = 

q+p e q+p 

a eee ee =(¢= 2p) | 
—(q-2+p)p q?-—2q¢+qp+1 |’ 


which, in view of the formula in Exercise 5.4, proves that P, = P?. 

We shall use induction to prove that P, = P" for alln € N. We already 
know that the assertion true for n = 1 (and also for n = 2). Suppose that 
P, = P”. Then some simple, but tedious algebra gives 


prti = pp” 
oe Ee Wares Ley eee 5 oe ey Weel yy 
= hey q I ap + ap DO)” tag ap a) 
mr 
P 1-4) se ae PO” at pl — P49)” 
se ga) ere enemies, aay (cere ee al 
n n 
arp ap —P-9) gp t qep\l -—P—9) 


= Pati: 
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Solution 5.7 


Recall that P?+® = P”P*. Since pni4(j|i) are the entries of the matrix Pa+z = 
prtk = Pp’ P* we obtain (5.18) directly from the definition of the product of 


two matrices. 


Solution 5.8 


Since the 7; are independent, €, and n+ are also independent. Therefore, 


PGi = s|£o =18 050 nea = Sn) 
= P(&_+n41 = s|£o = $0,°'+,€n = Sn) 
= P(m+1 = 8 — 8nl€o = 80,°**,€n = $n) 


= P( 41 =8— Sn). 
Similarly, 


P (Ena = slfn = Sx) Pen ee = s\f, = Sn) 
ae cf (M41 = 5 — Salon = Sn) 
= P(n+1 =$— Sn). 


Solution 5.9 


Step 1. For n = 1 the right-hand side of (5.19) is equal to 0 unless |j —7| < 1 
and 1+ 7 —1 is even. This is only possible when 7 = 1+ 1 or 7 = 1-1. In 
the former case the right-hand side equals p, and in the latter it equals g. This 
proves (5.19) for n = 1. 
~~ Step 2. Suppose that (5.19) is true for some n. We will use the following 
version of the total probability formula. If H; € F, P(H; H;) = 0 fori F j, 
and P(U, Hi) = 1, then 


P(A|C) = S_ P(A|C 0 Hi) P(H,|C). (5.67) 


Then the Markov property and (5.67) imply that 


P(En+1 = j|&o = 1) 

= P(Er4i = jo =t,6n =7 —1)P(&m = 35 — 1 bo = 7) 
+P(En41 = Jo =t,€n =J + 1)P(En = 5 + 1]o = 2%) 
P(En4i = Jn =I —1)P(En = 3 — 1 fo = 2) 
+P(En4i = 5|€n =F +1)P(E& =J + 110 = 1) 


_ n ntj-l-i n-j+l4i n nt+j+l—i n—j—1ti 
= P\ ntj-1-t JP * 4% FE nt jei-i |P 
2 2 


i 
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uv n eg 
= (Geen) + (ners) | p= qd 
2 2 


( n+l n4+1+j-i n+1-j+i 
2 q 2 : 


n+1+j-1 
2 


Solution 5.10 
Denote the right-hand side of (5.24) by h(a). It follows by induction that 


| 1 
ni (ey = PMG gay A, Jol cE 


On the other hand, h is analytic and 
a (k) k = 
ha)= > a (O)x*, |z| < " 


Solution 5.11 


We shall translate the problem into the Markov chain language. Denote by S$ 
the set of all natural numbers N = {0,1,2,---}. Let €, denote the number of 
males in the nth year (or generation), where the present year is called year 0. If 
En =1, ie. there are exactly 71 males in year n, then the probability that there 
will be 7 males in the next year is given by 


P (Ena = jf, =1) = P(X, +--+ X; = 7), (5.68) 


where (X,)72, is a sequence of independent identically distributed random 
variables with common distribution 


P(X =m) = Pm, meEN. 
Hence €,, n > 0 is a Markov chain on S with transition probabilities 


Notice that p(0|0) = 1, ie. if €, = 0, then £, = 0 for all m > n. Dying out 
means that eventually €, = 0, starting from some n € N. Once this happens, 
€, will stay at O forever. 


Solution 5.12 


We shall only deal with part 2), as in part 1) there is nothing to show. Suppose 
that £9 = 7. Then & = Xi + ---X;, where X; are independent identically 
distributed Poisson random variables with parameter A. Since the sum of such 
random variables has the Poisson distribution with parameter 2, (5.27) follows 
readily. 
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Solution 5.13 


Let €,, denote the nurnber of supporters at the end of day n. Then €,41 — &p is 
equal to 1 or 0 and 


Pq; ifj=i+l, 
plt)=< l1-pq,  ifj =1, (5.70) 
0, otherwise. 


Thus €, is a Markov chain on a state space S = N with transition probabilities 
p(j|t) given by (5.70). 


Solution 5.14 


We shall use the notation introduced in the hint. Observe that €,41 — £n — Zn 
equals —1 or 0. The latter case occurs with probability p, i.e. when the car 
served at the beginning of the nth time interval was finished by the end of the 
nth time interval. The former case occurs with probability 1 — p. Therefore 


\M-ttl \i-t 
P a = 47 a ape tae P= ace 
(fn4i = Jfn = 2) PGFs il + ( PG He 
for 2 > 1,7 >1—1. On the other hand, if 7 > 0, then 
I 
Pl a = 76 = 0) = Fil x 
Thus, €, is a Markov chain with transition probabilities 
qj» if 1=0,7 EN, 
P(J|t) = 4 hind ifi>lj>i-1, (5.71) 
0, otherwise, 
where gq, = ee and 


' PQks if k 
= 5.72 
z (1 — p)qe+i + par, if k 0202) 


The transition probability matrix of our chain takes the form 


[ go qa O 0 


nan 4 % O 
Q2 &@ © gm J 
P= ; ! 


q2 qh 
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Solution 5.15 


If p,(j|t) > 0, then P(€n = 7 for some n > Ol) = 7) > pe(Jlt) > 0. If pe (gl) = 
for all k > 1, then 


P(, = j for some n > O|fo =#) < D> P(E = jlo = 4) = D_ Pals) = 0 
n=1 n=1 


Solution 5.16 


Since P(f = 2|€& =72) = 1, it follows that 7 4 7, which proves 1). Assertion 2) is 
obvious. To prove 3) we proceed as follows. From the solution to Exercise 5.15 
we can find n,m > 1 such that p,(j|i) > 0 and pm(klj) > 0. Hence, the 
Chapman-Kolmogorov equations yield 


Pmn(Kli) = Y> pm(k|S)Pn(s|é) > Pm(Klj)Pa(Jlé) > 
se€S 


Solution 5.17 


Since |pp(j|i)| < 1 and |fn(j|t)| < 1 for ali n € N, the radii of convergence of 
both power series are > 1. To prove the equalities (5.40)—(5.41) we shall show 
that for n > 1 and any i,j € S, 


S77 (Jl) pax (91d). (5.73) 


k=1 


By total probability formula and the Markov property 
Pr(glt) = P(E = jl =?) 


= Pia j a7 F51 Sha 1=)) 


k=1 
= SOP& =56 #5151 <k- 16 =1) 
k=1 
xP(En =jl&e =F #F1<1<k-1) 


= Di GHPExH alee =) 
— S AUmrOb): 
k=1 


Solution 5.18 
Since 0 < S pals 1) < land 0 < fn(jli) < 1, the result follows readily from 


AT _139_ TL 
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Solution 5.19 


Only the case of a recurrent state needs to be studied. Suppose that 7 is recur- 
rent. Then \>>”, fa(|Z) = 1. Hence, by Exercise 5.18, Fj;(z) Alasz 1. 
Thus P;;(x) = (1 — Fj;(z))~* - c0 as x 11, and so, again by Exercise 5.18, 
>, <1 Pn(J|j) = co. Conversely, suppose that )>°”_, pn(j|j) = oo. Then, by 
Exercise 5.18, P;;(z) = oo asx 7.1. Thus, Fj;(x) = 1 — (P;;(x))~* 7 1 as 
x 71. Hence, 3-7, fa(jl7) = 1, which proves that j is recurrent. 

To prove (5.43) we use (5.73) to get 


S— pa(dlt) 
| -& 


co n-l 


> \- fn—e(J|t)Pe (Jl) 


n=0 k=0 

S> > fm(ili)pe Sa) 
k=0 m=1 

So pe (ils) 3° fm le) 
k=0 m=1 

S— px (13). 

k=0 


This implies (5.43) when j is transient. 


n==0 


lI 


IA 


Solution 5.20 


We shall show that the state 0 is recurrent. The other case can be treated in 
a similar way. From Exercise 5.8 we have pp(0|0) = >} + 52 (1—p-—q)”. 
Thus pn (0|0) + 54, > 0, and so > 9 Pn(0|0) = 00, which proves that 0 is a 
recurrent state. 


Solution 5.21 


Suppose each j7 € S is transient. Then by (5.43) 0) pra(jlt) < for all 
i € S. Let us fix i € S. Then we would have })j.5 0724 Pn(jt) < 00, 
since S is finite. However, this is impossible because ies 1 Palit) = 


eae, jeg Pn(Jlt) a eee 1 = ©. 
Solution 5.22 


Let us begin with a brief remark concerning the last part of the problem. Since 
the random walk is ‘space homogenous’, i.e. p(j|z) = p(j—7|0), it should be quite 
obvious, at least intuitively, that either all states are positive-recurrent or all 
states are null-recurrent. One can prove this rigorously without any particular 
difficulty. First, observe (and prove by induction) that pr(j|i) = pr(j — 2/0). 
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Second, observe that the same holds for fy, ie. fr(j|t) = fr(g — 2/0). Hence, in 
particular, mj; = mo. 
To show that 0 is null-recurrent let us recall some useful tools: 


Poo(t) = )/pn(0|0)e", -1 <2 <1, 
n=0 

Foo(z) = D/ fa(0|0)a", -1<2< 1. 
n=1 


Since, see Exercise 5.9, 


por (0|0) = Gr 
Pai) = (2) @) ial 


Then, using (5.41), we infer that Foo(z) = 1—(1—27)!/?. Since F(z) 7 00 as 
a 7 1 and Fo9(z) = S77, nfn(0|0)z” by using Abel’s lemma (compare with 
Exercise 5.18), we infer that }°°°., nfn(0|0) = oo. This shows that mo = oo 
and thus 0 is null-recurrent. 


Solution 5.23 


We know that 0 is a recurrent state. From definition 
fr(0|0) = p(0|1)p(1|1)"~7p(1|0) = pg(1 — g)"~?. 


Since |1 — g| < 1, we infer that $7), nfn(O0|0) = 0°, pq(1 —q)"? < ow. 
Hence mo < oo and 0 is positive-recurrent. The same proof works for state 1. 


Solution 5.24 


Consider a Markov chain on S = {1,2,---,9} with transition probabilities given 
by the graph in Figure 5.3. Then, obviously, p4(1|1) = 1/2 and pg(1|1) = 1/2, 


Figure 5.3. Transition probabil- 
ities of the Markov chain in Exer- 
cise 5.24 


but p,(1]1) = 0 if k < 6 and k ¢ {4,6}. Hence d(1) = 2, but po(1|1) = 0. 
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Solution 5.25 


We begin with finding P2 = P? in an algebraic way, i.e. by multiplying the 
matrix P by itself. We have 


2 _| 0 1/2 0 1/2 | _ 
a-P=(T elt ia] 

Alternatively, P; can be found by observing that the only way one can get from 
1 to 1 in two steps is to move from 1 to 2 (with probability 1) and then from 2 
to 1 (with probability 1/2). Hence, the probability p2(1|1) of going from 1 to 1 
in two steps equals 1/2. Analogously, we calculate p2(1|2) by observing that in 
order to move from 1 to 2 in two steps one needs first to move from 1 to 2 (with 
probability 1) and then stay at 2 (with probability 1/2). Hence, po(1|2) = 1/2. 
The remaining two elements of the matrix Pz can be found by repeating the 
above argument, or, simply by adding the rows so that they equal 1. In the 
latter method we use the fact that P, is a stochastic matrix, see Exercise 5.3. 
The graph representing FP is shown in Figure 5.4. 


dole bole 
lO 


1/2 


1/2 3/4 


Figure 5.4. Two-step transition 


1/4 
: probabilities in Exercise 5.25 


Using any of the rnethods presented above, we obtain 


| 


Therefore, pi(1|1) = 0, po(1{1) = 1/2 and p3(1|/1) = 1/4. Hence d(1) = 1 
(although p;(1|1) = 0). Since p; (2/2) = 1/2 > 0, it follows that d(2) = 1. 


n= 


ml |e 
00/07100/09 


Solution 5.26 


Suppose that ) = 7. Denote by 7 the minimum positive time when the chain 
nters state k, i.e. 


feo= Minh Ss 6 = KY}: 
, P{t; < mi} =: > Oifi > j andi # j. If 7 » i, then it would 
be impossible to return to 2 with probability at least e > 0. But this cannot 
happen as 2 is a recurrent state. Indeed, 
l= P<) =P < Ol <4) PG <7) 
+P(1% < colt; 2 7:1) P(t; > Ti). 
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The second term on the right-hand side is < 1—€ < 1, while the first factor in 
the first term is equal to 0 (since i » 7). This is a contradiction. The second 
part is obvious. 


Solution 5.27 


As observed in the hint, we only need to show that (5.45) holds when p(j|z) = 0 
fori € C and j € S\ C. We begin with the following simple observation. If 
(92, F, P) is a probability space and A, € F,n EN, then P(U,, An) = 0 if and 
only if P(A,) = 0 for each n € N. Hence (5.45) holds if and only if for each 
k>n 
P(&ES\ClEn, € C) =0. (5.74) 

In fact, the above holds if and only if it holds for k = n + 1. Indeed, suppose 
that for each n € N 

Pigaa €S \ Cle, eC) =0. (5.75) 
Let us take n € N. We shall prove by induction on k > n that (5.74) holds. This 
is so for k =n and k = n+ 1. Suppose that (5.74) holds for some k > n + 1. 
We shall verify that it holds for k + 1. By the total probability formula (5.67) 
and the Markov property (5.10) 


P (Greve S \Cicn-€'C) 
=P(Exr1 ES \ClEn € CLE EC) P(E € ClEn € C) 
+P (Ext € S\Cl&n € C,d E S\C) P(& €S\ Clé& € C) 
= P (En41 € S\Cl& € C) P(E € Clfn € C) 
+P (ke+1 ES \Cl&e ES \C)P(heES\ClEn € C). 
By the induction hypothesis P(E, € S\C|€, € C) = 0 and by (5.75) (applied to 
k rather than n) P (41 € S\ Cl& € C) = 0. Thus, P (41 € S\ Cl&, € C) = 
0, which proves (5.74). 
The time-homogeneity of the chain implies that (5.75) is equivalent to (5.74) 


for n = O. Since P is a countably additive measure and S is a countable set, 
the latter holds if and only if p(j|t) =O forte C andj E S\C. 


Solution 5.28 


Property 2) in Proposition 5.8 implies that pw is an invariant measure. Therefore 
it remains to prove uniqueness. Suppose that v = es qjo; 1S an invariant 
measure. It is sufficient then to show that 7; = q; for all 7 € S. 

Since 0 < pa(jlt)g:i < q for all 1,7 € S and )),q; = 1 < ow, Lebesgue’s 
dominated convergence theorem yields 


Le @) 
qj = S— pnlJli)ai > ars = 7. 
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It follows that p = v. 


Solution 5.29 


We shall argue as in the uniqueness part of Solution 5.28. If v = ae qj0; is 
an invariant measure, then by Lebesgue’s dominated convergence theorem 


CO 
a5 = > pPaldlilas + © 154: = 0. 
i=l i 
Hence yv = 0, which contradicts the assumption that v is a probability measure. 


Solution 5.30 


Obviously, P = , ; | . Therefore equation Pa = 7 becomes 
Tm = 72, 
TQ = Ty. 


The only solution of this system subject to the condition 7; + m2 = 1 is m = 
2 = 1/2. 


Solution 5.31 


Put = Diieg Hid; and v = D7 ,-5%46;. Then, for any j € S andneN 


d_ palsli) (1 — 8) pi + v4) = (1-9) D> padi) oe + 9D pn (Sli)v% 
iE€S iEeS ieS 
(1 — 6)p;j + Ov; = [(1—0)p + 6y),. 


Solution 5.32 
Put “= doiceg Hidi. Then, for any j € T 


Hy = >> paljlé)ui > 0 
iE€S 


by the Lebesgue dominated convergence theorem. Indeed, by Exercise 5.19 
Pn(j|t) 2 0 for alli € S, and pa(j|t); < wi, where 5> pi < oo. 


Solution 5.33 


Since 7 is transient in view’of (5.43), from Exercise 5.19 we readily get (5.53). 


Solution 5.34—— 
By Theorem 5.5 pn(j|t) > a for all i,j € S. Put 7; = aie € S. We need to 
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hea = ae m;6; is an invariant measure of the chain €,, n € N; 
2) aw is the unique invariant measure. 


Part 1) will follow from Proposition 5.8 as soon as we can show that 7; > 0 for 
at least one 7 € S. But if 7 € S, then 7 is positive-recurrent and thus m; < oo. 
Part 2) follows from Exercise 5.28. 


Solution 5.35 


Let us fix 1,7 € Z. First suppose that j —2 = 2a € 2Z. Then poz44(j|i) = 0 for 
all k € N, and also po,(j|t) = 0 if k < |a|. Moreover, if k > |a|, then 


Gkcaf 22h oe eee. ye kk 
palit) = (,°°)P q - (4) kta)? ¢- 


Since (,7*,) < (2), it follows that 


k+a 
a ° (2k 
por (J|t) < (2) ( ) pa +0 
q k 


by Proposition 5.4. We have therefore proven that 7; is well defined and that 
nm; = 0 for all 7 € Z. Hence, we infer that no invariant measure exists. 


Solution 5.36 


Suppose that 7 = paren 7j;0; iS an invariant measure of our Markov chain. 
Then )°, p(j|t)a7i = 7; for all j € S=N. Using the exact form of the transition 
probability matrix in the solution to Exercise 5.14, we see that the sequence 
of non-negative numbers (7;)92o solves the following infinite system of linear 
equations: 


GoTo + 997 = To 
Mm + 4) ™ + qoT = 7 (5.76) 
q2To + 97 + 9,72 +9973 = Te 
1.€. 
k+1 
GTO + > Oba —7Ti = Ths KEN. 
j=l 


Multiplying the kth equation in (5.76) by z*, k > 0, and summing all of them 
up we obtain 


G(x) 


x 


toQ(x) + S10) = I(x), |2| <1, (5.77) 
j=l 
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where for |z| < 1, 


IT(z) = So aya ; 
7=0 

Q(z) = oF: 
j=O0 

G(r) = > gx? 
j=0 


Since et m;21 = II(x) — 7, we see that IT, Q and G satisfy the functional 

equation 

G(r) — Q(z) 
G(x) -2 

Since all the coefficients 7; in the power series defining JJ are non-negative, 

Abel’s lemma implies that 


IT(x) = mo lz] <1. (5.78) 


It is possible to use (5.78) to calculate this limit. However, there is still some 
work be done. We have to use |’Hospital’s rule. Since G(x) — z > 0, G(z) — 
tQ(r) > 0, G'(r) —1 4 N' — 1 and (G(z) — 2Q(z))’ 3  —X=— 1 (all limits 
are for x / 1), we obtain (recall that 4’ = lim, », G'(z)) 
1) if \’ #1, then H(z) > mo 4454; 

2) if \’ = 1, then I(r) - oo. 

In case 1) 14A—X > 1, since mp < 1. It follows that ay > 0, and so A’ < 1. 
Moreover, in this case 


1—.)' 
™ = >——~.. 
pales be oh | 
The above argument shows that if there exists an invariant measure, then it is 


unique. 

Now suppose that an invariant measure exists (and then it is unique). Since 
our chain is irreducible and aperiodic (check this!), it follows from Theorem 5.6 
that the Markov chain €, is ergodic. Therefore, by Theorem 5.5, for any 7 € N 


Pn(Jjlj) 4 fj, aS — 0. 
Since F;;(1) = 1, it also follows that 


Pn(jlt) 4 7;, asn— ov. 


/ 
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Solution 5.37 


We begin with the case no = 1. Then M,(j) — mn(7) < (1 — €)” from (5.62). 
It follows that 


Mn (J) 
Mn (J) 


IAN IA 


Pn(Jlt) < Maly), 


and we infer that |pp(j|t) — 7;| < (1 -—e«)", n € N. Suppose that np > 2. Then 
for any r=1,---,no—1 


Pknotr(Jlé) — 731 = |S Pano (ils) Pr (slé) — D> tipr(slé) 


ses sES 
< Y- [Peno (ils) — malPr (sli) 
seS 
< max |Peno(Jls) — 73] D_ Pr(slé) < (1—€)”. 
sES 


Solution 5.38 


The transition probability matrix P takes the form | : a 1 q ; | Since 
all four numbers p,q,1— p,1—q are strictly positive, the assumption (5.54) 
is satisfied, and so the limits 7; = lim, pn(j|t) exist and are i-independent. 
Hence, the unique invariant measure of the corresponding Markov chain is 
equal to 79d9 + 7161. (Recall that S = {0,1} in this example.) We need to find 
the values 7;. One way of finding them is to use the definition. As in Hint 5.28, 
the vector 7 = (70,7) solves the following linear equation in matrix form: 


es mn thd ee (5.79) 
mo +m = 1. (5.80) 


Some elementary algebra allows us to find the unique solution to the above 
problem: 


| Pp 
t= —, 71 = —. 5.81 
o p+q 0 op+q ey 


Hence, —-6) + —* 6, is the unique invariant measure of the Markov chain. 


> pt+q 
Solution 5.39 


From (5.66) we infer that 


p+q 


Steet (haggis 


POO) = ee 
" q+p q+t+p q+p 
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Pp Pp Pp 
PaO) SS pg) a 
| q+p qtp q+ p 
q q n q 
PAC Le ee SS LP a) 
| q+p qtp qt+p 
Pp q n Pp 
Dal |) ee eS Peg) ee 
q+p qtp q+p 
Hence mm = ee and 7, = aaa: This is in agreement with the previous solution. 


Solution 5.40 


Suppose that €, = 71. Then the value of €,4, depends entirely on the outcome of 
the next roll of a die, say X n+, and the value of €,,. Since it depends on the past 
only through the value of €,, intuitively we can see that we are dealing with a 
Markov chain. To define €, in a precise way consider a sequence of independent 
identically distributed random variables X,, n = 1,2,3,--- such that P(X, = 
1) = 1/6 for alli = 1,2,---,6. Then, putting €,41 = max{€,, Xn41}, we can see 
immediately that P(€,41 = in+1|€0 S45, ae Ha) PCE ia = inti lEn = 
tn) and 


P(Enti =J5ln = 2) = PlEnti = 3m =1,Xng1 < 1)P(Xngi <2) 
+P(En+1 a HN€n = 1, Xn41 ? t)P(Xn41 ? i) 


i ; ae Saif ms if j >i, 
0 ifj <i, ? es 
se if j >i, 

i 
0 ifj <i. 

Thus, 
ae oe 
PUI)=4 gE 7 =7, 
0) if 7 < 4. 
It follows that 

zx 0 0 0 0 0 
1 2 
ae ee 
a A 2: 10 @ 

Pes a A 
6 @ 6 ¢ UO O 
i ar ee Cae Cane are 
6 6 6 6 6 
ary ee eae ae ae 
6 6 6 6 6 


Solution 5.41 


The eranh af the chain is given in Figure 5.5. The transition matrix is 
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Figure 5.5. Transition probabil- 
ities of the Markov chain in Exer- 
cise 5.41 


0) 

|| 
eee ete eri 
Hl JK AlroO CGC 
Ale Aiwo CG © 
Kr OO° 


Either from the graph or from the matrix we can see that 1 > 3 if and only if 
i<j. The state i = 1 is transient. Indeed, p(1|1) = 1/4 and p,(1|1) = (1/4)” 
by induction. Therefore, >), pn(1|1) < co and 1 is transient by Exercise 5.19. 
The same argument shows that the states 2 and 3 are also transient. On the 
other hand, p(4|4) = 1, and so 4 is a positive-recurrent state. (As we know, 
that there should be at least one positive-recurrent state.) 

We shall find invariant measures by solving the system of four linear equa- 
tions Px = am for m = (71,72,73,74), Subject to the condition 7, + m2 + 
73 + 74 = 1. Some elementary linear algebra shows that the only solution is 
7 = 7. = 73 = 0, m4 = 1. Thus, the unique invariant measure is 7 = 64. 

The invariant measure can also be found by invoking Theorem 5.4. In our 
case C' = {4}, and so there is exactly one class of recurrent sets. Moreover, as 
we have seen before, 4 is positive-recurrent. Therefore, there exists a unique 
invariant measure. Since its support is contained in C’, we infer that ma = 64. 


Solution 5.42 


The graph representing the Markov chain for a = 4 is presented in Figure 5.6. 
Obviously, i > j fori € S \ {0,a} =: S and j € S. Moreover, 0 > j if and 
only if j = 0 and a — j if and only if 7 = a. Since p(0|0) = p(ala) = 1, 
both states 0 and a are positive-recurrent. All other states are transient. For 


° 
if 2 € S were recurrent, 7 would be intercommunicating with 0 because 2 — 0, 
hv Exercise 5.26. This is impossible. Therefore. by Remark 5.5 there exist an 
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Figure 5.6. Transition probabil- 
ities of the Markov chain in Exer- 
cise 5.42 


infinite number of invariant measures: uy = (1 — 6)d9 + Ada, where @ € (0, 1]. 
Indeed, 5, is the only probability measure with support in the singleton {a}. 
It is possible to verify this with bare hands. Next, let (7) denote the probabil- 
ity that the investigated Markov chain €, hits the right-hand barrier prior to 
hitting the left-hand one. Once €, hits the left barrier it will never leave it, so 
g(i) is actually the probability that €, hits a. Put 


A= {3n€N:&, =a}. 


Then, by the total probability formula and the Markov property of £,, we have 
forl<i<a-l1 


(i) = P(AlG = 4) = > P(Alé =4,8 = J) P(& = jl6o =) 


j=0 
= So P(A|& = 9) P(E = Slo = 4) 
j=0 
= >) (9) P(E = Jl = i) = poi + 1) + ad(é - 1). 
j=0 
Obviously, 
¢(0) = 0, 
g(a) = 1. 


Therefore, the sequence (¢(i))‘_9 satisfies 


Qo 

aN 

=. 

wT 
| 


= po(i+1)+qe(2-1), 1<i<a-l, (5.82) 
¢(0) = 0, g(a) = 1. 


Since p + q = 1, equations (5.82) can be rewritten as follows: 


P{d(t + 1) - o()) =4[¢@) -@-D], 1siga-1. 


[ 
1 
: 


fensetent tenting att A NEE FEE RO PT EF EY BEEN 
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Hence, | 
[oi + 1) — o(t)] = (4) x, 


where zx = (1) — $(0) is to be determined. Using the boundary condition 
g(a) = 1, we can easily find that 


1=¢(a) = S> (oi +1) - $(@)) 


Here we assume that p £ q. The case p = q = 1/2 can be treated in a similar 
way. In fact the latter is easier. It follows that 


: —1 
anal (2)¢-1 
and therefore 
ee (4y'-1 
O(7) = D_ (o(k +1) — (k)) = = 
(oa 
k=0 p 


ee 
» 


6 


Stochastic Processes in Continuous Time 


6.1 General Notions 


The following definitions are straightforward extensions of those introduced 
earlier for sequences of random variables, the underlying idea being that of a 
family of random variables depending on time. 


Definition 6.1 


A stochastic process is a family of random variables €(t) parametrized by t € T, 
where T C R. When T = {1,2,...}, we shall say that €(t) is a stochastic process 
in discrete time (i.e. a sequence of random variables). When T is an interval 
in R (typically T = (0,00)), we shall say that €(t) is a stochastic process in 
continuous time. 

For every w € {2 the function 


Tatr &t,w) 


is called a path (or sample path) of €(t). 


Definition 6.2 


ily F; of o-fields on (2 parametrized by t € T,, where T C R, is called a 
filtrationif 


Fs, CF: CF 


far anv oe t C\T anch that oc < t 
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Definition 6.3 


A stochastic process €(t) parametrized by t € T is called a martingale (sub- 
martingale, supermartingale) with respect to a filtration F; if 


1) €(t) is integrable for each t € T; 


2) €(t) is Fz-measurable for each t € T (in which case we say that €(t) is 
adapted to F;); 


3) €(s) = E (E€(t)|F,) (respectively, < or >) for every s,t € T such that s < t. 
In earlier chapters we have seen various stochastic processes, in particular, 
nartingales in discrete time such as the symmetric random walk, for example. 


n what follows we shall study in some detail two processes in continuous time, 
1amely, the Poisson process and Brownian motion. 


3.2 Poisson Process 


3.2.1 Exponential Distribution and Lack of Memory 


definition 6.4 


Ne say that a random variable 7 has the exponential distribution of rate A > 0 


f 
P{n>th=e* 


or all t > 0. 


For example, the emissions of particles by a sample of radioactive material 
or calls made at a telephone exchange) occur at random times. The probability 
hat no particle is emitted (no call is made) up to time t is known to decay 
xponentially as ¢ increases. t_is to say, the time 77 of the first emission has 
he exponential distribution, P {n > t}=e- 


-xercise 6.1 


Vhat is the distribution function of a random variable n with exponential 
istribution? Does it have a density? If so, find the density. 


(int What is the probability that 7 > 0? What is the probability that 7 > t for any 
iven t < 0? Can you express the distribution function in terms of P {7 >t}? Is the 
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Exercise 6.2 
Compute the expectation and variance of a random variable having the expo- 


nential distribution. 


Hint Use the density found in Exercise 6.1. 


Exercise 6.3 


Show that a random variable 7 with exponential distribution satisfies 
P{n>t+s}=P{n>t}P {n> s} (6.1) 
for any s,t > 0. 


Hint When the probabilities are replaced by exponents, the equality should become 
obvious. 


Exercise 6.4 


Show that the equality in Exercise 6.3 is equivalent to 
P{n>t+s|n >s}=P{n>t} (6.2) 
for any s,t > 0. 


Hint Recall how to compute conditional probability. Observe that 7 > s + ¢ implies 
n> S. 


The equality (6.2) (or, equivalently, (6.1)) is known as the lack of memory 
property. The odds that no particle will be emitted (no call will be made) in 
the next time interval of length ¢ are not affected by the length of time s it has 
already taken to wait, given that no emission (no call) has occurred yet. 


Exercise 6.5 


Show that the exponential distribution is the only probability distribution sat- 
isfying the lack of memory property. 


Hint The lack of memory property means that g(t) = P {n > t} satisfies the func- 
tional equation 

g(t + s) = g(t)g(s) 
for any s,t > 0. Find all non-negative non-increasing solutions of this functional 
equation. 
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6.2.2 Construction of the Poisson Process 


Let 7,,72,..- be a sequence of independent random variables, each having the 
Same exponential distribution of rate ’. For example, the times between the 
emissions of radioactive particles (or between calls made at a telephone ex- 
change) have this property. We put 


Ex =™ +--+, 


which can be thought of as the time of the nth emission (the nth call). We also 
put > = 0 for convenience. The number of emissions (calls) up to time t > 0 is 
an n such that &,41 >t > &,. In other words, the number of emissions (calls) 
up to time t > 0 is equal to max {n: t > &,}. 


Definition 6.5 


We say that N(t), where t > 0, is a Poisson process if 


N(t) = max{n:t > &)}. 


Thus, V(t) can be regarded as the number of particles emitted (calls made) 
up to time ¢. It is an example of a stochastic process in continuous time. A 
typical path of N(t) is shown in Figure 6.1. It begins at the origin, N(0) = 0 
(no particles emitted at time 0), and is right-continuous, non-decreasing and 
piecewise constant with jumps of size 1 at the times €,. 


N(t) 


1 


l. 


—-— 
2 m—<—<- 
1f re | 
0 1 | | Ho 


Figure 6.1. A typical path of N(t) and the jump times €, 
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What is the distribution of N(t)? To answer this question we need to recall 
the definition of the Poisson distribution. 


Definition 6.6 


A random variable v has the Poisson distribution with parameter a > 0 if 
a” 

= a —a 
P{v=n}=e oy 


for anyn=0,1,2,.... 


The probabilities P {v = n} for various values of a are shown in Figure 6.2. 


0123 ”" 012345 7" 0123456789 %n 


Figure 6.2. Poisson distribution with parameter a 


Proposition 6.1 


N(t) has the Poisson distribution with parameter At, 


Proof 


First of all, observe that 


{N(t) <n} = {& >t}. 
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It suffices to compute the probability of this event for any n because 
P{N(t)=n} = P{N(t) <n+1}- P{N(t) <n} 
P {Engi > t} — P {Em >t}. (6.3) 


We shall prove by induction on n that 


n—1 k 
Pi Sipe} Oe (6.4) 


For n= 1 
P{é, >t}=P{m>t}=e. 


Next, suppose that (6.4) holds for some n. Then, expressing £,,) as the sum 
of the independent random variables €, and nni1, we compute 


P {En41 > t} = P {En + Mn41 > t} 
P {tnt > th +P {Er > t—Mn41,t > M41 > OF 


t 
a ey | P {€, >t —s} fn4,(s) ds 
0 


1 k 
(A(t=5))*, sg, 


o 
| 
» 
fon 
ae 
o~——~ 
fo 
fav) 
| 
> 
< 
| 
el 


= k! 

n-1 \k41 et 

= e *! 467% . (t—s)*ds 
k! 
k=0 
n k 
—\t (At) 
- 2S ki? 
k=0 


where f,,,,,(8) is the density of M,41. By induction, (6.4) holds for any n. Now 
apply (6.3) to complete the proof. O 


Exercise 6.6 


What is the expectation of N(t)? 


Hint What are the possible values of V(t)? What are the corresponding probabilities? 
Can you compute the expectation from these? To simplify the result use the Taylor 
expansion of e”. 


Exercise 6.7 
Compute P {N(s) = 1, N(t) = 2} for any0<s <t. 


Hint Express {N(s) = 1, N(t) = 2} as {m <s<m+m<t<m+m2+73}. You 
can compute the probability of the latter, since 1, 72, 73 are exponentially distributed 
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6.2.3 Poisson Process Starts from Scratch at Time t 


Imagine that you are to take part in an experiment to count the emissions of 
a radioactive particle. Unfortunately, in the excitement of proving the lack of 
memory property you forget about the engagement and arrive late to find that 
the experiment has already been running for time ¢t and you have missed the 
first N(t) emissions. Determined to make the best of it, you start counting right 
away, so at time t+s you will have registered N(t+s) — N(t) emissions. It will 
now be necessary to discuss N(t + s) — N(t) instead of N(s) in your report. 

What are the properties of N(t+s)—N(t)? Perhaps you can guess something 
from the physical picture? After all, a sample of radioactive material will keep 
emitting particles no matter whether anyone cares to count them or not. So 
the moment when someone starts counting does not seem important. You can 
expect N(t + s) — N(t) to behave in a similar way as N(s). And because 
radioactive emissions have no memory of the past, N(t + s) — N(t) should be 
independent of N(t). 

To study this conjecture recall the construction of a Poisson process N(t) 
based on a sequence of independent random variables 7), 72,... , all having the 
same exponential distribution. We shall try to represent N(t +s) — N(t) ina 
similar way. 

Let us put 


mh = En(t)41 =f, nh = 7N(t)tn, Ls 2, 3, Caps 
see Figure 6.3. These are the times between the jumps of N(t+s)— N(t). Then 
we define 
f= Mts + Mh 
= max{n: & < s}. 


€n,-1 Ew t Ent Ew.42 En.43 


Figure 6.3. The random variables nj,n2,75,... 
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Exercise 6.8 


Show that 
N‘'(s) = N(t+s) — N(t). 


Hint First show that €4 = Enit)in — t. 


If we can show that 7£,7{,... are independent random variables having the 
same exponential distribution as 7, 72,... , it will mean that N(t +s) — N(t) 
is a Poisson process with the same probability distribution as N(s). Moreover, 
if the 7, turn out to be independent of M(t), it will imply that N(t+s)— N(t) 
is also independent of N(t). 

Before setting about this task beware of one common mistake. It is some- 
times claimed that the times ni, 7{,7$,... between the jumps of N(t+s)— N(t) 
are equal to £n41 — t-n+2;7n+43,--- for some n. Hence nj, 7%,7{,... are inde- 
pendent because the random variables €n41,n4+2,n+3,--- are. The flaw in this 
is that, in fact, nf, 75,n{,... are equal to €,41 —t, Wn42, Mn43,--- only on the set 
{N(t) = n}. However, the argument can be saved by conditioning with respect 
to N(t). Our task becomes an exercise in computing conditional probabilities. 


Exercise 6.9 


Show that 
P{ni > 8|N(t)} = P{m > s}. 


Hint It suffices (why?) to verify that 
P{ni > 8, N(t) =n} = P{m > 8} P{N(t) =n} 
for any n. To this end, write the sets {N(t) = n} and {nj > s, N(t) = n} in terms of 


En and n+1, which are independent random variables, and use the lack of memory 
property for 7n41. 


Exercise 6.10 
Show that 
P{m > 81,---5m > seIN(t)} = P{m > 81}---P{ne > se}. 
Hint Verify that 
P{ni > $1,m > 82,---,Mk > sk, N(t) =n} = P{m > s1}---P{m, > sk}P{N(t) =n} 


for any n. This is done in Exercise 6.9 for k = 1. 
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Exercise 6.11 


From the formula in Exercise 6.10 deduce that the random variables 7, and 
N(t) are independent, and the 7’, have the same probability distribution as np, 
i.e. exponential with parameter 4. 


Hint What do you get if you take the expectation on both sides of the equality in 
Exercise 6.10? Can you deduce the probability distribution of n’,? Can you see that 
the 7* are independent? 

To prove that the nf, are independent of N (t) you need to be a little more careful 
and integrate over {N(t) = n} instead of taking the expectation. 


Because N(t + s) — N(t) can be defined in terms of nj,7§,... in the same 
way the original Poisson process N(t) is defined in terms of 7,72,... , the 
result in Exercise 6.11 proves the theorem below. 


Theorem 6.1 
For any fixed t > 0 
N‘(s) = N(t+s)—N(t), s>0 
is a Poisson process independent of N(t) with the same probability law as N(s). 
That is to say, for any s,t > 0 the increment N(t+s) — N(t) is independent 


of N(t) and has the same probability distribution as N(s). The assertion can be 
generalized to several increments, resulting in the following important theorem. 


Theorem 6.2 
For any 0 < ft) < tg <--- < t, the increments 
N(t1), N(tg) — N(ti), N(ts) — N(t2),..., (tn) — N(tn-1) 


are independent and have the same probability distribution as 


(t,). N(te — t1), N(t3 —tz),..., N(ta —tn—1)- 


Proof 
From Theorem 6.1 it follows immediately that each increment N(t;) — N(tj-1) 
has the same distribution as N(t; —t;_1) fori =1,...,n. 


It remains to prove independence. This can be done by induction. The case 
when n = 2 is covered by Theorem 6.1. Now suppose that independence holds 
for n increments of a Poisson process for some n > 2. Take any sequence 


Not Ctra oOo... ot St... 
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By the induction hypothesis 
N(tn41) — N(tn),...,N(t2) — N(fi) 
are independent, since they can be regarded as increments of 
N"®(s) = N(t, +s) — N(t1), 


which is a Poisson process by Theorem 6.1. By the same theorem these incre- 
ments are independent of N(t,). It follows that the n + 1 random variables 


N(tn41) ~ N(tn); oe , N(t2) _ N(t1), N(t1) 


are independent, completing the proof. O 


Definition 6.7 


We say that a stochastic process €(t), where t € T, has independent increments 
if 


f(t) = € (to) c<b (ta) Se (tne) 
are independent for any to < ty <--- < ty such that to,ti,...,th ET. 


Definition 6.8 


A stochastic process €(t), where t € T’, is said to have stationary increments if 
for any s,t € T the probability distribution of €(t + h) — €(s + h) is the same 
for each h such that s+h,t+heT. 


Theorem 6.2 implies that the Poisson process has stationary independent 
increments. The result in the next exercise is also a consequence of Theorem 6.2. 


Exercise 6.12 


Show that N(t) — At is a martingale with respect to the filtration 7; generated 
by the family of random variables {N(s) : s € [0, t]}. 


Hint Observe that N(t) — N(s) is independent of 7; by Theorem 6.2. 


6.2.4 Various Exercises on the Poisson Process 


Exercise 6.13 


Show that  < & <&2 <--- as. 
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Hint What is the probability of the event {€n-1 < &n} = {nn > 0}? What is the 
probability of the intersection of all such events? 


Exercise 6.14 


Show that limn_+o9 En = 00 as. 


Hint If limn+oo&n < oo, then the sequence m,72,... of independent random vari- 
ables, all having the same exponential distribution, must be bounded. What is the 
probability that such a sequence is bounded? Begin with computing the probability 
P{m <™m,...,1n < m} for any fixed m > 0. 

Although it is instructive to estimate P {limn+oo&n < oo} in this way, there is a 
more elegant argument based on the strong law of large numbers. What does the law 
of large numbers tell us about the limit of Sn as nN — co? 


Exercise 6.15 


Show that €, has absolutely continuous distribution with density 


(At)"—} 


fr(t) = Neo (n—1)! 


with parameters n and A. The density f,,(t) of the gamma distribution is shown 
in Figure 6.4 for n = 2,4 and A= 1. 


igure 6.4. Density f,(t) of the 
, 4 3 t mma distribution with parame- 
tersn=2,AX=landn=4,A=1 


Hint Use the formula for P {€, >t} in the proof of Proposition 6.1 to find the dis- 
tribution function of €,. Is this function differentiable? What is the derivative? 


Exercise 6.16 


Prove that 
jim N(t) =00 as. 


150 Basic Stochastic Processes 


Hint What is the limit of P{N(k) > n} as k > oo? Can you express {limt—oo N(t) 
= oo} in terms of the sets {N(k) > n}? 


Exercise 6.17 


Verify that 


P{N(t) is odd} = e~*' sinh(At), 
P{N(t) is even} = e~*‘ cosh(At). 


Hint What is the probability that N(t) = 2n +1? Compare this to the n-th term of 
the Taylor expansion of sinh At. 


Exercise 6.18 


Show that 
N(t) 


lim ——— = A 
t 
if N(t) is a Poisson process with parameter 4. 


Hint N(n) is the sum of independent identically distributed random variables N(1), 
N(2) — N(1),..., N(n) —- N(n—1), so the strong law of large numbers can be applied 
to obtain the limit of N(n)/n as n + oo. Because M(t) is non-decreasing, the limit 
will not be affected if n is replaced by a continuous parameter t > 0. 


6.3 Brownian Motion 


Imagine a cloud of smoke in completely still air. In time, the cloud will spread 
over a large volume, the concentration of smoke varying in a smooth manner. 
However, if a single smoke particle is observed, its path turns out to be ex- 
tremely rough due to frequent collisions with other particles. This exemplifies 
two aspects of the same phenomenon called diffusion: erratic particle trajec- 
tories at the microscopic level, giving rise to a very smooth behaviour of the 
density of the whole ensemble of particles. The Wiener process W(t) defined 
below is a mathematical device designed as a model of the motion of individual 
diffusing particles. In particular, its paths exhibit similar erratic behaviour to 
the trajectories of real smoke particles. Meanwhile, the density fy t) of the 
random variable W(t) is very smooth, given by the exponential function 


2? 


e at, 


3] - 


r)= 
fw) (2) er 
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which is a solution of the diffusion equation 


Of 10°f 
Ot 2 dx? 
and can be interpreted as the density at time t of a cloud of smoke issuing form 
single point source at time 0. The Wiener process W(t) is also associated with 
the name of the British botanist Robert Brown, who around 1827 observed 
the random movement of pollen particles in water. We shall study mainly the 
one-dimensional Wiener process, which can be thought of as the projection of 
the position of a smoke particle onto one of the axes of a coordinate system. 
Apart from describing the motion of diffusing particles, the Wiener process 
is widely applied in mathematical models involving various noisy systems, for 
example, the behaviour of asset prices at the stock exchange. If the noise in the 
system is due to a multitude of independent random changes, then the Central 
Limit Theorem predicts that the net result will have the normal distribution, a 
property shared by the increments W(t) — W(s) of the Wiener process. This is 
one of the main reasons of the widespread use of W(t) in mathematical models. 


6.3.1 Definition and Basic Properties 


Definition 6.9 


The Wiener process (or Brownian motion) is a stochastic process W(t) with 
values in R defined for t € [0, 00) such that 


1) W(0) = Oas:;; 

2) the sample paths t » W(t) are a.s. continuous; 

3) for any finite sequence of times 0 < t; < -:: < ty, and Borel sets 
Ayj,...,4n CR 


P{W(ti) € A1,...,W(tn) € An} 


-/ Le (t,,0,21) p(t2 —t1,21,22) °° 
Ay 


p(tn = tn 1,;2n-1; In) dz, -- -dZn, 
where 1 ey)? 
t,z,y) = Ee 3t 6.5 
p(t,z,y) or (6.5) 


defined for any x,y € R and t > 0 is called the transition density. 


A tywnieal camnie nath af the Wiener nrocess is shown in Figure 6.5. 
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Figure 6.5. A typical path of 


W (t) 
Exercise 6.19 
Show that 
l 22 


fwt) (x) a Jant 


is the probability density of W(t) and find the expectation and variance of W(t). 


Hint The density of W(t) can be obtained from condition 3) of Definition 6.9 written 
for a single time ¢ and a single Borel set. You will need the formula 


+00 22 
i e 2?dr=vJ2r 


Lo.@) 


to compute the integrals in the expressions for the expectation and variance. 


Remark 6.1 

The results of Exercise 6.19 mean that W(t) has the normal distribution with 
mean 0 and variance t. 

Exercise 6.20 


Show that 
E (W(s)W(t)) = min {s,t}. 


Hint The joint density of W(s) and W(t) will be needed. It can be found from 
condition 3) of Definition 6.9 written for two times s and t and two Borel sets. 


Exercise 6.21 


Show that 
E (IW (t) = W(s)!") =|t—s|. 
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Hint Expand the square and use the formula in Exercise 6.20. 


Exercise 6.22 
Compute the characteristic function FE (exp (iAW (t))) for any AE R. 


Hint Use the density of W(t) found in Exercise 6.19. 


Exercise 6.23 

Find E (W(t)*). 

Hint This can be done, for example, by expressing the expectation in terms of the 
density of W(t) and computing the resulting integral, or by computing the fourth 


derivative of the characteristic function of W(t) at 0. The second method is more 
efficient. 


Definition 6.10 


We call W(t) = (Wi(t),...,W(t)) an n-dimensional Wiener process if 
W(t),...,W™(t) are independent R-valued Wiener processes. 


Exercise 6.24 


For a two-dimensional Wiener process W(t) = (W1(t),W?(t)) find the prob- 
ability that |W(t)| < R, where R > 0 and |z| is the Euclidean norm of 
¢ = (a!,27) in R’, ie. |z|* = ()° + (0?)*. 


Hint Express the probability in terms of the joint density of W'(t) and W?(t). In- 
dependence means that the joint density of W'(t) and W?(t) is the product of their 
respective densities, which are known from Exercise 6.19. It is convenient to use polar 
coordinates to compute the resulting integral over a disc. 


6.3.2 Increments of Brownian Motion 


Proposition 6.2 


For any 0 < s < t the increment W(t) — W(s) has the normal distribution with 
mean Q and variance t — s. 
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Proof 
By condition 3) of Definition 6.9 the joint density of W(s), W(t) is 
fw(s),W(t) (a, y) ceo, (3, 0, x) p (t — §,2, y) : 


Hence, for any Borel set A 


P{W(t) —W(s) € A} = (s,0,2) p(t —s,z,y) drdy 
{(z,y):y— a 


| Rs es 
| t-s,0,w) au) dx 
A 

+00 


= [ pe-s.0. au | p(s,0,x) dx 
A 


—co 


= [ pt-s,0,w du. 
A 


But f(u) = p(t —s,0,u) is the density of the normal distribution with mean 
O and variance t — s, which proves the claim. 0 


Corollary 6.1 


Proposition 6.2 implies that W(t) has stationary increments. 


Proposition 6.3 
For any 0= to < ty < -+- <t, the increments 
W(t) — W(to),.--;W (tn) — (tea) 


are independent. 


Proof 


From Proposition 6.2 we know that the increments of W(t) have the normal 
distribution. Because normally distributed random variables are independent 
if and only if they are uncorrelated, it suffices to verify that 


E [(W(u) — W(t) (W(s) —W(r))] =0 
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for any 0 <r<s<t<u. But this follows immediately from Exercise 6.20: 


E [((W(u) — W(t)) (W(s) — W(r))] 


E(W(u)W(s)) — E(W(u)W(r)) 
— E(W(t)W(s)) + BE (W(t)W(r)) 
SPS Ser 

0, 


as required. U 


Corollary 6.2 


For any 0 < s < t the increment W(t) — W(s) is independent of the o-field 


F, =o {W(r):0<7r<s}. 


Proof 


By Proposition 6.3 the random variables W (t)—-W(s) and W(r)—W (0) = W(r) 
are independent if 0 < r < s < t. Because the o-field #, is generated by such 
Wir), it follows that W(t) — W(s) is independent of 7,. U 


Exercise 6.25 


Show that W(t) is a martingale with respect to the filtration F;. 


Hint Take advantage of the fact that W(t) — W(s) is independent of F; if s < t. 


Exercise 6.26 
Show that |W (t)|? —#is a martingale with respect to the filtration F;. 
Hint Once again, use the fact that W(t) — W(s) is independent of F; if s < t. 


Let us state without proof the following useful characterization of the 
Wiener process in terms of its increments. 


Theorem 6.3 


A stochastic process W(t),t > 0, is a Wiener process if and only if the following 
conditions hold: 


1) W(0) = O0as:.; 


2) the sample paths t +> W(t) are continuous a.s.; 
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3) W(t) has stationary independent increments; 


4) the increment W(t) — W(s) has the normal distribution with mean 0 and 


variance t — s for any0<s<t. 


Exercise 6.27 
Show that for any T > 0 


V(t) =W(t+T) -W(T) 
is a Wiener process if W(t) is. 


Hint Are the increments of V(t) independent? What is their distribution? Does V(t) 
have continuous paths? Is it true that V(0) = 0? 


The Wiener process can also be characterized by its martingale properties. 
The following theorem is also given without proof. 


Theorem 6.4 (Lévy's martingale characterization) 


Let W(t),t > 0, be a stochastic process and let 7; = o(W,,s < t) be the 
filtration generated by it. Then W(t) is a Wiener process if and only if the 
following conditions hold: 

1) W(0) =Oas.; 
2) the sample paths t + W(t) are continuous a.s.; 


) 
) 

3) W(t) is a martingale with respect to the filtration F;; 
) 


Exercise 6.28 
Let c > 0. Show that V(t) = 4W(c*t) is a Wiener process if W(t) is. 


(t)|? — t is a martingale with respect to F;. 


= 


4 


Hint Is V(t) a martingale? With respect to which filtration? Is |V(t)|? —¢ a martin- 
gale? Are the paths of V(t) continuous? Is it true that V(0) = 0? 


6.3.3 Sample Paths 


Let 
C= i <i Se ee ST. 
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where 
wT 
aa 


be a partition of the interval [0,7] into n equal parts. We denote by 
APW = W(tR,,) - W(t) 


the corresponding increments of the Wiener process W(t). 


Exercise 6.29 


Show that : 
° n 2 = . 2 
jim z (ArVW)* =T in L’. 


Hint You need to show that 
n-l1 2 
: n 2 ae 
lim E ([ (A?W) | = 0. 


Use the independence of increments to simplify the expectation. What are the expec- 
tations of A?W, (A?W)? and (A?W)*? 


The next theorem on the variation of the paths of W(t) is a consequence of 
the result in Exercise 6.29. First, let us recall that the variation of a function 
is defined as follows. 


Definition 6.11 
The variation of a function f : [0,T] + R is defined to be 


n—-l 


sup dX \f(tita) — F(ta)I, 


where t = (to, t1,...,tn) is a partition of [0,T], i.e. O=to <tr <-:- <tr =T, 
and where 
At= max ; [tina — t|. 


1=0;..4n 


Theorem 6.5 


The variation of the paths of W(t) is infinite a.s. 
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Proof 
Consider the sequence of partitions t? = (t?,t?,...,#") of [0,7] into n equal 
parts. Then 

n-1 os 

> ae (m_, ar) » |ArW|. 


Since the paths of W(t) are a.s. continuous on [0,7], 


lim ( max 1420) =0 as. 


noo \i=0,...,n 
By Exercise 6.29 there is a subsequence t™* = (t9*,t]*,...,¢7*) of partitions 
such that 
ne—1 
lim Am™W|\?=T as. 
k—+00 » | ‘ | 


This is because every sequence of random variables convergent in L? has a 
subsequence convergent a.s. It follows that 


np—l 


lim S- |Ay*W|=oco as., 
k— oo =p 


while 


T 
lim At"* = lim — =O, 
k—+00 k—+oo Np 


which proves the theorem. 0 


Theorem 6.5 has important consequences for the theory of stochastic inte- 
grals presented in the next chapter. This is because an integral of the form 


T 
i f(t) dW (t) 


\ 
N 


cannot be defined pathwise (that is, separately for each w € §2) as the Riemann- 
Stieltjes integral if the paths have infinite variation. It turns out that an intrinsi- 
cally stochastic approach will be needed to tackle such integrals, see Chapter 7. 


Exercise 6.30 
Show that W(t) is a.s. non-differentiable at t = 0. 
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Hint By Exercise 6.28 V.(t) = -W(c’t) is a Wiener process for any c > 0. Deduce 
that the probability 


p {mol > cM for some t € (0, -z]} 
is the same for each c > 0. What is the probability that the limit of We exists as 
t \, 0, then? 
Exercise 6.31 


Show that for any ¢ > 0 the Wiener process W (t) is a.s. non-differentiable at t. 


Hint Vi(s) = W(s +t) — W(t) is a Wiener process for any t > 0. 


A weak point in the assertion in Exercise 6.31 is that for each t the event of 
measure 1 in which W’(t) is non-differentiable at t may turn out to be different 
for each t > 0. The theorem below, which is presented without proof, shows 
that in fact the same event of measure 1 can be chosen for each t > 0. This is 
not a trivial conclusion because the set of t > 0 is uncountable. 


Theorem 6.6 
With probability 1 the Wiener process W(t) is non-differentiable at any t¢ > 0. 


6.3.4 Doob’s Maximal L? Inequality for Brownian Motion 


The inequality proved in this section is necessary to study the properties of 
stochastic integrals in the next chapter. It can be viewed as an extension of 
Doob’s maximal L? inequality in Theorem 4.1 to the case of continuous time. 
In fact, in the result below the Wiener process can be replaced by any square 
integrable martingale £(t), t > 0 with a.s. continuous paths. 


Theorem 6.7 (Doob’s maximal L? inequality) 


For any t > 0 
E (ax¥7(0)P < 4E|W(t)|?. (6.6) 
gS 


Proof 


For t > 0 and n € N we define 


kt 
up =|w (=) 0< ke a" (6.7) 
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Then, by Jensen’s inequality, Mj’,k = 0,---,2", is a non-negative square in- 
tegrable submartingale with respect to the filtration F7? = F ke, SO by Theo- 


rem 4.1 
B (max Mp?) < 4z|Mp = az) (0)? 


Since W(t) has a.s. continuous paths, 


lim max |M?|? = max|W(s)|?_ as. 
noo k<2" s<t 


Moreover, since Mf = Mjj,*", the sequence sup, <n |Mj|?, n € N, is increasing. 
Hence by the Lebesgue monotone convergence theorem max,<;:|W(s)|? is an 
integrable function and 


k< 


B (maxi ()P) = Jim £ (max iMer) <4zIW OP, 


completing the proof. 0 


6.3.5 Various Exercises on Brownian Motion 


Exercise 6.32 
Verify that the transition density p(t, z, y) satisfies the diffusion equation 


Op _ 13'p 
Ot 2dy? 


Hint Simply differentiate the expression (6.5) for the transition density. 
Exercise 6.33 
Show that Z(t) = —W(t) is a Wiener process if W(t) is. 


Hint Are the increments of Z(t) independent? How are they distributed? Are the 
paths of Z(t) continuous? Is it true that Z(0) = 0? 


Exercise 6.34 
Show that for any0<s<t 


P{W(t) € A|W(s)} = [ p(t — 8, W(s),y) dy. 
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-Hint Write the conditional probability as the conditional expectation of 14(W(t)) 
given W(s). Compute the conditional expectation by transforming the integral of 
la(W(t)) over any event in the o-field generated by W(s). This can be done using 
the joint density of W(s) and W(t). Refer to the chapter on conditional expectation 
if necessary. 


Exercise 6.35 


Show that e“(e-2 is a martingale. (It is called the exponential martingale.) 


Hint What is the expectation of e““)-WS) for 5 < t? By independence it is equal 
to the conditional expectation of e~“-W*) given F,. This will give the martingale 
condition. 


Exercise 6.36 
Compute FE (W(s)|W(t)) forO<s<t. 


Hint You want to find a Borel function F such that EF (W(s)|W(t)) = F (W(t)), ie. 


/ W(s)dP = i: F(W(t)) dP. 
{W(t)E A} {W(t)EA} 


Either side of this equality can be transformed using the joint density of W(t) and 
W(s). 


6.4 Solutions 


Solution 6.1 


Suppose that 7 is a random variable with exponential distribution of rate 1. 
The distribution function of 7 is 


0 ift <0, 
F()=Pinst}=1-Pr>d={ 0 ow ites 


Therefore 7 has density 


d 0 if t < 0, 
AO ar Net if t > 0. 
The distribution function F(t) and density f(t) are shown in Figure 6.6. 


Solution 6.2 


Tloine the doncitw f(t) = \e—At fannd in Exercise 6.1 and integrating bv parts. 
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Figure 6.6. The distribution function F(t) and density f(t) of a random variable 
with exponential distribution of rate A = 2 


we obtain 
E(n) = / t f(t) dt = i thet dt = -{ t—e' dt 
—0o 0 0 dt 
ste Pe +/ e—“' dt =0- set = - 
0 0 0 r 


In a similar way we compute 
E(n?) 2 Py jat= | Pre at = — | 29 6M at 
—0o (8) 0 dt 
2 


= fe |" +2 | te dt 045 | tf(t)dt = —. 
0 0 0 


2 


It follows that the variance is equal to 
var (9) = E (n?) - (E(m))” = 53 - 3 = 


Solution 6.3 


By the definition of a random variable with exponential distribution 


Pigs s+ih-SeOrn 2ae-%e* = P{n>s}P{n>t}. 


Solution 6.4 


By the definition of conditional probability 
P{n>t+s,n>s} 
P {n> s} 

_ Pin>t+s} 
—  P{n>s} ? 


Il / 


P{n>t+sln>s} 
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- Solution 6.5 


The exponential distribution is the only probability distribution satisfying the 
lack of memory property because the only non-negative non-increasing solutions 
of the functional equation 


g(t + s) = g(t)g(s) 


are of the form g(t) = a’ for some 0 <a <1. 
To verify this observe that [g(m/n)]” = g(m) = [g(1)]™ for any integers m 
and n #0. Let a := g(1). It follows that 


g(q) =a!’ foranygEQ 


Since g is non-increasing, 0 <a < 1 and 
t 


t ‘ 
a* = inf > g(t) > su =a, 
LCL a 9( ) al ee 9(q) 


so indeed 
g(t) =a’ foranyteR 


As aresult, P{n > t} = a’ for some 0 < a < 1. But the distribution function 
of a random variable cannot be constant, so 0 4 a # 1. Hence a = e~* for some 
A > 0, completing the argument. 


Solution 6.6 


Since N(t) has the Poisson distribution with parameter At we have E (N(t)) = 
At. Indeed 


E(N(t)) = 5 nP{N(t) =n}= Yo nen 0 AN" 
= Me™ Y aan 7 Mert ert — Mt. 


Solution 6.7 


Using the fact that 7,,2,... are independent and exponentially distributed, 


we obtain 
P{N(s) =1,N(t)=2} = P{liss<&<t< &} 


= P{m<s<mt+m<t<mt+m+73} 


& 
i P{s<utm <t<u+ne+73} r\e A"du 
0 


s t—u 
[ ( P{t<utv+yns} red) re" du 
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Solution 6.8 


Since 


it follows that 


Solution 6.9 
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Ss t—u 
_ / (/ eNEw8) ,e Ady Ae >“ du 
0 s—U 


= \2e%*s (¢ — 8). 


te ae ca 
= Enct)t4i — b+ 7N(t)42 +°°° + NN(t)4n 


= En(t)tn — f, 


N'(s) = max{n: & < s} 
max{n : En(t)tn St + 5} 


max{n:&, <t+s}-— N(t) 
= N(t+s)—N(t). 


It is easily verified that 


{N(t) =n} = {Nn41 >t — En, t > En} 


{nj >s,N(t) =n} = {Nn41 >st+t—-—&,t>&} 


Since &n,n41 are independent and nn+1 satisfies the lack of memory property 
from Exercise 6.3, 


P {nj >s,N(t)=n} = P{mt1 >st+t—€n,t > En} 


t 
= / P{nn41 >s+t—u} Pe, (du) 


— oo 


t 
P{tmsi > 8} P {nner > tu} Pe, (du) 
—- C&O 


P {Mn+41 > s} P {n41 >t— En, t 2 En } 
P {tsi > s} P{N(t) =n} 
P{m >s}P{N(t) =n}. 


The last equality holds because nn4; has the same distribution as 7,. Now 
divide both sides by P{N(t) = n} to get 


P{nj > s|N(t) =n} = P{m > 3} 
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for any n= 0,1,2,.... As a result, 
P{nj > s|N(t)} = P{m > s} 


because N(t) is a discrete random variable with values 0,1, 2,.... 


Solution 6.10 
As in Solution 6.9, 
{N(t) = n} = {Mn41 >t— Enis t 2 ays 
{ni > s1,N(t) =n} = {nngi > S1+t—-&n,t > En} 
and, more generally, 
{nj > 81,75 > S2,...,1% > 8, N(t) =n} 
cea {Nn41 >s,+t— En, t 2 fn} M {Nn+2 > So} MN: A {Nn+k > Sx}. 


Since €n,%n+1,---;%n+k are independent and 7n+2,...,%n4% have the same 
distribution as 7,...,7,, using Exercise 6.9 we find that 


P{ni > $1,175 > 82,..., > 8x, N(t) =n} 
= P{nnti > sitt—€ni€n < t}P{tnz2 > $2}-::Pitinsk > 8k} 
= P{n > s1,N(t) = n}P{n2 > s2}---P{ne > sx} 
= P{ni > s1|N(t) = n}P{n2 > s2}-+-P{n, > se}P{N(t) = n} 
= P{m > 81}P{n2 > s2}---P{n, > 4} P{N(t) = n}. 

As in Solution 6.9, this implies the desired equality. 


Solution 6.11 
Take the expectation on both sides of the equality in Exercise 6.10 to find that 
P{m > 81,---, > 8k} = P{m > s1}---P{ne > sx}. 
If all the numbers s,, except perhaps one are zero, it follows that 
Pig Seah SPS Sale = Lyng hk, 


so the random variables 7’, have the same distribution as 7,. Inserting this 
back into the first equality, we obtain 


P{ni > $15.6, > sp} = P{ni > s:}---P{nj > sx}, 


so the 7, are independent. 
To prove that the 7’, are independent of N(t) integrate the formula in 
Exercise 6.10 over {N(t) = n} and multiply by P{N(t) = n} to get 


P{ni > 81,...,7% > 8k, N(t) =n} = P{m > s1}---P{n, > sp} P{N(t) =n}. 
But P{nt > sn} = P{nn > 8n}, hence N(t) and the n*, are independent. 
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Solution 6.12 


We need to verify conditions 1), 2), 3) of Definition 6.3. Clearly, N(t) — At is 
F-measurable. By Exercise 6.6 


EF (|N(t)|) = E(N(t)) = At < 00, 


which means that N(t) is integrable, and so is N(t) — At. 
Theorem 6.2 implies that N(t)—N(s) is independent of F, for any 0 < s < t, 
SO 


E(N(t) — N(s)|F,) = E(N(t) — N(s)) = E(N(t)) — E(N(s)) = At — As. 
It follows that 
E(N(t) — dt|Fs) = E(N(s) — As|F) = N(s) — As, 
completing the proof. 


Solution 6.13 
Since Mn = £n — €n-1 and P{n, > 0} = e® = 1, 


Pes <6 <6 <-} =P (Yim >0}) =1 


n=l 


Here we have used the property that if P(A,) = 1 for all n = 1,2,... , then 
P(()o Aa) =. 


Solution 6.14 


Since 
(oe) 
faa 
n= 
it follows that 


{ lim €, < 00 } C {m,72,... is a bounded sequence} 
n— oo 
(ee) (oe) 
= U fim sm) 
m=1n=1 — 


Let us compute the probability of this event. Because {tn <m}, N= 
1,2,... is a contracting sequence of events, 


r (Aiw<m) = Jn.2 (Aim mt 


n= n=l 
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N 
dim, TT Pen <m} 


= lim (ieee) 
Noo 


0. 


It follows that 


P(ume<a) <°(0 Aimsm) 
< S>P (A <m)) 
m=1 n=1 
= 0, 


completing the proof. 
While it is instructive to work through the above estimates, there exists a 
much more elegant argument. By the strong law of large numbers 
1 

Jim. a 
Here + is the expectation of each of the independent identically distributed 
random variables n, (see Exercise 6.2). It follows that 

lim £;=0oO 45., 

n— oo 


as required. 


Solution 6.15 


In the proof of Proposition 6.1 it was shown that 


n—1 k 
P {é, >t} = a oe 
k=0 ° 


for t > 0, see (6.4). Therefore the distribution function 
Oe 


F,(t) = P {& St}=1-P{& >t} =e Do 
of €, is differentiable, the density f, of £, being 
frlt) = =Fr(t) 
enw es oo + Ne 


k=n 
re) aie 
(n —1)! 


= Ae 
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for t > 0, and clearly f,(t) = 0 for t < 0. 


Solution 6.16 


Because N(t) has non-decreasing trajectories 


{ lim N(t) = oo} = A) LJ EN (k) > n}. 


n=1k=1 


Also, {N(k) > n}, k = 1,2,... is an expanding sequence of events and 


P{N(k) >n} = e™ 3 - 


i=0 
[t follows that 2 
p{ Um > “f =i, 
k=1 
30 ee 
P { Jim N(t) = oof =p ' | U {N(k) > a} =e 
n=1 k=1 
Solution 6.17 
Since 
et —e-% oe g2ntl 
sinh(z) = i eas 2 Qn +1)!’ 
em +e77 a“ een 
cosh(z) = 5 » (Qn)! 
ve have 


P{N(t) is odd} = 3 P{N(t) = 2n+1} 


n=0 
_ 3 éM (\t)2741 
Nc (2n + 1)! 


= e~*' sinh(At), 


P{N(t) is even} = 3 P{N(t) = 2n} 


n=0 


eae At)?" 
=e (2n)! 


n=0 


e~** cosh(At). 
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Solution 6.18 


We can write 
N(n) = N(1) + (N(2) — N(1)) +--+ + (N(n) - N(n -1)), 


where NV (1), N(2) — N(1), N(3) — N(2),... is a sequence of independent iden- 
tically distributed random variables with expectation 


By the strong law of large numbers 
lim ae) =X: AS. (6.8) 
m—> OO nr 
Now, ifn<t<n+1, then N(n) < N(t) < N(n +1) and 
N(n) . N(t) — N(n +1) 


< —— < ———_—_.. 
n+1l7- ¢t 7 n 


By (6.8) both sides tend to A as n > oo, implying that 


Solution 6.19 
Condition 3) of Definition 6.9 implies that 


fw (x) = p(t, 0, 2) 


is the density of W(t). Therefore, integrating by parts, we can compute the 
expectation 


+00 
E(W(t)) = [ zp(t,0,r) dx 


2 


1 ie : 
= —— re 2dr 
JV 2nt fie 
t MOO - s2 
== i —e 2dr 
V2rt J_g dz 


t 22 [too 
_ — e@. 2 = 0 


V27t —0o 
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nd variance 


E ((w(t))”) - [- x p(t,0,x2) dz 


—oo 


= a [Pet as 
2nt J = 
t es d wie 
=e | rae 2 dz 
t _ gz jtoo t mee, ee 
“om er --# 
t ee ye 
= 0+ — e 2du=t 
V20 I-00 


Ve have used the substitution u = Me and the formula stated in the hint. 


‘olution 6.20 


uppose that s < t. Condition 3) of Definition 6.9 implies that the joint density 
[ W(s) and W(t) is 


fw(s),w(t) (x,y) = P (s, 0,2) p(t — 8, r,y) : 


follows that 


+00 +00 
E(W(s)W(t)) = / ye ry p(8,0,2)p(t— 8,2,y) de dy 


+00 +00 
=| xp(s,0,2) ( f yp(ts,2,) dy) da 


+00 
= / z*p(s,0,xr) dx = s. 


—0o 


his is because by the results of Exercise 6.19 


ie a 
yp(t—s,2,y) dy = f(z +u) p(t-s,2,2+u) du 
+00 
=| (c+u) p(t—s,0,u) du 
+00 BB 
= a p(t — s,0,u) aut | up(t—s,0,u) du 
=2£+0=2 
id 


+00 
/[ z’°p(s,0,z) dz =s. 
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It follows that for arbitrary s,t > 0 

E (W(s)W(t)) = min {s,t}. 
Solution 6.21 
Suppose that s < t. Then by Exercise 6.20 


E (|W (t) = W(s)/?) = E(W(t)?) —2E(W(s)W(t)) + E (W(s)?) 
= t-—2@s+s=t-s. 


In general, for arbitrary s,t > 0 
E (|W (2) = W(s)|?) = |t—s|. 
Solution 6.22 


Using the density fw t) (x) = p(t, 0,2) of W(t), we compute 


+0O 
E (exp (iAW(t))) = / e?n(t,0,2) dz 


—OO 
1 c et At o— dr 
V2Tt J—oo 
1 _ ate i? (2=iae)? 
= e 2 e dx 
V 27t as 
rt 


Solution 6.23 


Using the formula for the characteristic function of W (t) found in Exercise 6.22, 
we compute 


d* 
E(W(t)') = <4] Blexp @aw(o)) 
=0 
= __ e 2 
dr4 |) 5 
= 3t? 


Solution 6.24 
Since W!(t), W?(t) are independent, their joint density is the product of the 
densities of W!(t) and W?(t). Therefore 


P{\W(t)| < R} = i cam P(tOs#) P(t Osu) de dy 
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Qmt 


1 R 2n 2 
=o —% dpd 
waif, fy Te # ae 
Rd 2 


= —e i d 
[ = r 


2 
=1l-e *®. 


on 
- = | e~ =r dr dy 
{|z|<R} 


'e have used the polar coordinates R,y to compute the integral. 


olution 6.25 
wanyO<s<t 
E (W(t)|Fs) 


E (W(t) —W(s)|Fs) + E(W(s)|Fs) 
E (W(t) — W(s)) + W(s) 
= W(s), 


nce W(t)—W(s) is independent of ¥, by Corollary 6.2, W(s) is ¥,-measurable 
id E(W(t)) = E(W(s)) =0. 
olution 6.26 
wanyO<s<t 
E(W(t)"|F.) = B (\W(t) — W(s)/? Fe) + B(2W(t)W(s)|Fe) 
=i (W(s)?|Fs) 
= E(|W(t) - W(s))?) + 2W(s)B (W(t)|Fe) 


—W(s)* 
= t—s+2W(s)? — W(s)? 
= t-s+W(s)’, 


nce W(t) — W(s) is independent of F, and has the normal distribution with 
ean 0 and variance t — s, W(s) is ¥,-measurable, and W(t) is a martingale. 


follows that 
E (W(t)? —t|F,) = W(s)? — s, 


; required. 
olution 6.27 


or any 0 < tp < t; <-:: < t, the increments 
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of V(t) are independent, since the increments 
W(tn+T)- W(tn—-1 + T),---, W(t, +T) - W (to + T) 


of W(t) are independent. For any 0 < s < t the increment V(t) — V(s) has the 
normal distribution with mean zero and variance t—s, since W(t+T)-W(s+T) 
does. Moreover, the paths th V(t) = W(t+T) — W(T) are continuous and 


V(0) = W(T) —- W(T) = 0. 
By Theorem 6.3 V(t) is a Wiener process. 


Solution 6.28 
It is clear that V(0) = 4W(0) = 0 as. and the paths t+» V(t) = 4W(c?t) are 


— ec 
a.s. continuous. We shall verify that V(t) and \V(t)|? — t are martingales with 
respect to the filtration 


Ge = a {V(s):0<s<t} 
= a {W(c*s):0<s<t} 
= 0 {W(s):0<s<c't} 
= F24. 


Indeed, if s < t, then c?s < c?t, so 
1 
B(VOIG.) = B (LWP) Fa.) 


= 1B (W(Ct)|F2,) 


and 
B(IvinP ts.) = B(4IMen? - AF.) 
1 
= GE (|W (ce)? — t|Fe2s) 


= 5 (|W (c?s)|? — cs) 
\V(s)|? — §, 


since W(t) and \W (t)|? —t are martingales with respect to the filtration F;. It 
follows by Levy’s martingale characterization that V(t) is a Wiener process. 
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Solution 6.29 


Since the increments A?W are independent and 


E(A?W)=0, E ((arw)?) =-, E ((47w)*) = ae 


) =9(E(en a 


-Fal( APw)? - 


i=0 


it follows that 


| 


n-1 
> (ArW)’ -T 
i=0 


n-1 


, oT T? 
s 3 E ((4: W) ‘i -—E ((4; w)?) + =| 
M1372 972) 2] oT? 
= > So = ae ee 


as 7m — OO. 


Solution 6.30 


We claim that, with probability 1, for any positive integer n there is at € (0, =z] 
such that wt > n. This condition implies that W(t) is not differentiable at 
t= 0; 
Let us put 
A= { wie > n for some t € (0, 2]} 


By Exercise 6.28 i 
= 4 


is a Brownian motion for any n. Therefore 
4 

p{ rel > ah 
4 

is { hanes > n} 


P{|W(1)|>2}741 asn>o. 


P (An) 


IV 


Since A;, Ag,... is a contracting sequence of events, 


P (N An] = lim P(An) = 1, 
n=1 


which proves the claim. 
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Solution 6.31 


By Exercise 6.27 Vi(s) = W(s + t) — W(t) is a Wiener process for any t > 0. 
Therefore, by Exercise 6.30 V;(s) is a.s. non-differentiable at s = 0. But this 
implies that W(t) is a.s. non-differentiable at t. 


Solution 6.32 


Differentiating 
1 (y-2)? 
p(t, ty) = e- — b) 
27t 
we obtain 
0 y? —2yxr+27-t 
pp Plt 2 y) = aera 70 aareraaae ) gr) 
O r-y 
gy Piety) = —— Plt, 2, y), 
6? y? —2yx+ 27 -t 
Hyp h ey) a 43 ——— P(t, £9), 
Xe) 
Op  10°p 
Ot 2 dy?’ 


as required. 


Solution 6.33 


Clearly, Z(t) = —W/(t) has a.s. continuous trajectories and Z(0) = —W/(0) 
0 as. If W(t) has stationary independent increments, then so does Z(t) 
—-W(t). Finally, 


Z(t) — Z(s) = — (W(t) — W(s)) 


has the same distribution as W(t) -—W(s), i.e. normal with mean 0 and variance 
t — s. By Theorem 6.3 Z(t) is a Wiener process. 


Solution 6.34 
Let 0< s <t. Then 


/ 14(W(t)) dP = P{W(s) € B,W(t) € A} 
{W(s)EB} 


/ 7 p(s,0,2)p(t — s,2,y) de dy 
BJA 


[ (/ p(t — 8,2, y) iy) p(s, 0, x) dx 


— ([ — s,W(s), y) dy dP 


\I 
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or any Borel set B C R. It follows that 


P{W(t) € A|W(s)} = FE (a(W(t))|W(s)) = | pe-s.Woov dy. 


solution 6.35 


Ne shall prove that e“(e~-2 is a martingale with respect to the filtration Fy. 
dlearly, it is adapted to the filtration F;, since W(t) is. Let 0 < s < t. Because 
V(t) — W(s) is independent of F, and W(s) is ¥,-measurable, 


E (eM MF.) = EB (eWO-W Wo) 7, ) 
— W)p (ee | F.) 
= Wp (eww) ; 
Che increment W(t) — W(s) has the normal distribution with mean 0 and 


rariance t — s, so the expectation of eW(4)—W(s) is equal to 


+00 


E (ewe) = | e* p(t — s,0, x) dx 


—oOo 


t-—s tO 
= ¢2 / p(t — s,0,2 —t) dz 


t follows that 
E (ce 4 IF, ) = eV(se-$, 


't also follows that eW()e—2 is integrable. Therefore eW(')e—2 is a martingale. 


Solution 6.36 


set 0 < s < t. We are looking for a Borel function F such that E (W(s)|W(t)) = 
F(W(t)), i.e 


| W(s) dP = | F(W(t)) dP 
{W(t)EA} {W(t)EA} 


or any Borel set A in R. The integral on the right-hand side can be written as 


: F (W(t) dP = i F(y) p(t,0,y) dy 
{W(t)E€A} A 


and the integral on the left-hand side as 


[ s)dP = cei (s,0, x) p(t t—9,0,y) de dy 
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- using the expression for the joint density of W(s) and W(t) in Solution 6.20. 
Let us compute the inner integral: 


+00 oO fs 
/ rp(s,0,z)p(t — s,2,y) dz = plt,0,y) | op (= t 2 Sy) dx 


—0oo — CO 


Ss 


(To see that the first equality holds, just use formula (6.5) for p(t, x, y).) There- 
fore 
s 
[ wtsyaP = [Fyn (t,0,u)ay. 
{W(t)E A} A 


It follows that F(y) = $y, ie. 


B(W(s)|W(t)) = W(t). 


7 


It6 Stochastic Calculus 


One of the first applications of the Wiener process was proposed by Bachelier, 
who around 1900 wrote a ground-breaking paper on the modelling of asset 
prices at the Paris Stock Exchange. Of course Bachelier could not have called 
it the Wiener process, but he used what in modern terminology amounts to 
W(t) as a description of the market fluctuations affecting the price X(t) of 
an asset. Namely, he assumed that infinitesimal price increments dX(t) are 
proportional to the increments dW (t) of the Wiener process, 


dX(t) = odW(t), 


where o is a positive constant. As a result, an asset with initial price X (0) = z 


would be worth 
X(t) =xz+oa0W(t) 


at time t. This approach was ahead of Bachelier’s time, but it suffered from 
one serious flaw: for any t > 0 the price X(t) can be negative with non-zero 
probability. Nevertheless, for short times it works well enough, since the prob- 
ability is negligible. But as t increases, so does the probability that X(t) < 0, 
and the model departs from reality. 

To remedy the flaw it was observed that investors work in terms of their 
potential gain or loss dX (t) in proportion to the invested sum X(t). Therefore, 
it is in fact the relative price dX (t)/X(t) of an asset that reacts to the market 
fluctuations, i.e. should be proportional to dW (t), 


dX(t) =o X(t) dW(t). (7.1) 
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What is the precise mathematical meaning of this equality? Formally, it resem- 
bles a differential equation, but this immediately leads to a difficulty because 
the paths of W(t) are nowhere differentiable. A way around the obstacle was 
found by Ito in the 1940s. In his hugely successful theory of stochastic integrals 
and stochastic differential equations It0 gave a rigorous meaning to equations 
such as (7.1) by writing them as integral equations involving a new kind of 
integral. In particular, (7.1) can be written as 


x(jsate [ ximaweo, 
0 


where the integral with respect to W(t) on the right-hand side is called the Ité 
stochastic integral and will be defined in the next section. While at first sight 
one would expect the solution to this equation to be ze”), in fact it turns 


out to be 
A(t) = ceV(t) e~3 | 


which is the exponential martingale introduced in Exercise 6.35. The intrigu- 
ing additional factor e~? is due to the non-differentiability of the paths of the 
Wiener process. Clearly, if x > 0, then X(t) > 0 for all t > 0, as required in 
the model of asset prices. In the following sections we shall learn how to trans- 
form and compute stochastic integrals and how to solve stochastic differential 
equations. 

Throughout this chapter W(t) will denote a Wiener process adapted to a 
filtration F, and L? will be the space of square integrable random variables. 


7.1 It6 Stochastic Integral: Definition 


We shall follow a construction resembling that of the Riemann integral. First, 
the integral will be defined for a class of piecewise constant processes called 
random step processes. Then it will be extended to a larger class by approxi- 
mation. 

There are, however, at least two major differences between the Riemann 
and It6 integrals. One is the type of convergence. The approximations of the 
Riemann integral converge in R, while the It6 integral will be approximated by 
sequences of random variables converging in 2. The other difference is this. 
The Riemann sums approximating the integral of a function f : [0,T] - R are 


of the form ; 


S— f(si)(ti41 — ty), 


j=0 
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-where 0 = to < t) < -:- <t, =T and s; is an arbitrary point in [t;,¢)41] for 
each j7. The value of the Riemann integral does not depend on the choice of the 
points s; € [t;,t;41). In the stochastic case the approximating sums will have 
the form 


S— f (83) (W (tj41) — W(t). 
j=0 


It turns out that the limit of such approximations does depend on the choice of 
the intermediate points s; in [t;,t;4,]. In the next exercise we take f(t) = W(t) 
and consider two different choices of intermediate points. 

Exercise 7.1 


Let O=t5 <t? <---<t =T, where t= if be a partition of the interval 
(0, 7] into n equal parts. Find the following limits in L?: 


n—0o 4 


lim >> W (th) (W(t) — W(¢F)) 
j7=—0 
and 


n—- 00 4 


n—1 
lim $7 W(t341) (W(t41) — W(t) - 
=0 
Hint Apply Exercise 6.29. You will need to transform the sums to make this possible. 
The identities 
a(b—a) = (0? a) — 5 (a—8)*, 


b(b—a) = 


wl wole 


(b? — a’) + 5 (a —b)? 
may be of help. 


The ambiguity resulting from different choices of the intermediate points s; 
in each subinterval [¢;,¢;41] can be removed by insisting that the approxima- 
tions of the integrand should consist only of processes adapted to the underlying 
filtration F;. This amounts to taking s; = ¢; for each j. The choice is motivated 
by the interpretation of F;: the value of the approximation at ¢ may depend 
only on what has happened up to time ¢t, but not on any future events. 


Definition 7.1 


We shall call f(t),¢ > 0a random step process if there is a finite sequence 
of numbers 0 = tp < t) < ... < ty, and square integrable random variables 
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70;71)---;n—1 such that 


n—l 
t) = So my ltes.ts41) (0), (7.2) 
j=0 


where 7; is #;,-measurable for 7 = 0,1,...,n — 1. The set of random step 
processes will be denoted by Mien: 


Observe that the assumption that the n; are to be F;;-measurable ensures 
that f(t) is adapted to the filtration 7. The assumption that the n; are square 
integrable ensures that f(t) is square integrable for each t. Also, Méep is a 
vector space, that is, af + bg € Mie, for any f,g € Mie, and a,beER. 


Definition 7.2 


The stochastic integral of a random step process f € M Sop of the form (7.2) is 
defined by 


= >> nj (W(tj41) — W(t;)). (7.3) 
1=0 


Proposition 7.1 


For any random step process f € M, tee the stochastic integral J(f) is a square 
integrable random variable, i.e. I(f) € L*, such that 


B (unr) =e( [ieee at). 


Proof 


Let us denote the increment W(tj;41) —W(t;) by A;W and t;41 —t; by Ajt 
for brevity. Then 


E(A;W) =0 and E (A*W) = A;t. 


First, we shall compute the expectation of 


n-ln- 
Ar=>> Sore WAW = Spay + 25> nym AjW AW. 
7=0 k=0 k<j 


Since n; and 4;W are independent, 


E (rn? A2W) = E (n?) E(A?W) = E(n?) A;jt 


7. It6 Stochastic Calculus 183 


Ifk <j, then jn. AxW and A;W are independent, so 
E (njnk AjW AW) = E (nin AkW) E(AjW) = 0. 
Therefore 


E(I(f)) = > B() Ajt 


It follows that I(f) € L?, since no,m,...,7n-1 € L?. 
On the other hand, 


n-ln-l1 


15M Lie; ,t541) \(t) Lie. try) (t -¥ 5 Lie ;.tj41)(¢ 
j=0 k=0 


(fue (ny? at) = DFO) A 
E (re) =2(f sora), 


implying that 


This means that 


as required. O 


Exercise 7.2 


Verify that for any random step processes f,g € Méep 


Bs) = ( [sol at) | 


Hint Try to adapt the proof of Proposition 7.1. Use a common partition 0 = to < 
ti <---<t, in which to represent both f and g in the form (7.2). 
Exercise 7.3 


Show that J : M2 


Sep — L? is a linear map, ie. for any f,g € M3., and any 
a,BeER 


I(af + Bg) = al(f) + BI(9). 


Hint As in Exercise 7.2, use a common partition 0 = to < ti <--- < tp, in which to 
represent both f and g in the form (7.2). 
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The stochastic integral J(f) has been defined for any random step process 
f € M2.,. The next stage is to extend J to a larger class of processes by 
approximation. This larger class can be defined as follows. 


Definition 7.3 
We denote by M? the class of stochastic processes f(t),t > 0 such that 


E( [is at) Ze 


and there is a sequence f,, fo,...€ Mies of random step processes such that 
lim & (/ If (t) — fr(t)|? it) = 0. (7.4) 
11-4 0O 0 


In this case we shall say that the sequence of random step processes fy, fo,... 
approximates f in M?. 


Definition 7.4 
We call I(f) € L? the Ité stochastic integral (from 0 to oo) of f € M? if 


Jim E (II(f) - 1(fa)I’) = 0 (7.5) 


for any sequence f,, fo,... € Meo of random step processes that approximates 
f in M?, ie. such that (7.4) is satisfied. We shall also write 


| ~ f(t) aw) 


in place of I(f). 


Proposition 7.2 


For any f € M? the stochastic integral I(f) € L? exists, is unique (as an 
element of L?, i.e. to within equality a.s.) and satisfies 


B(\nP)=2([ wera). (7.6) 


Proof 


It will be convenient to write 


Wllve =e ([ Pat) and Unllze = VEO) 
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_ for any f € M? and 7) € L?. These are norms! in M? and L?, respectively. 
Let fi, fo,... € Mitep be a sequence of random step processes approximat- 


ing f € M?, ie. satisfying (7.4), which can be written as 
Jim: If as Fallage = 0. 


We claim that I(f,), [(f2),... is a Cauchy sequence in L”. Indeed, for any € > 0 
there is an N such that ||f — fallyye < § for all n > N. By Proposition 7.1 


[I (fm) —I(fn)|lp2 I ( fm — fa), 

lfm — frllage 

If — milage + IF — fall 
ee 

9 + 9 == 

for any m,n > N, which proves the claim. 

Because L? with the norm ||-||;2 is a complete space (in fact a Hilbert 
space), every Cauchy sequence in L has a limit. It follows that I(f,),I(f2),... 
has a limit in L? for any sequence f;, f2,... of random step processes approxi- 
mating f. It remains to show that the limit is the same for all such sequences. 
Suppose that fj, fo,... and g1,g2,... are two sequences of random step pro- 
cesses approximating f. Then the interlaced sequence fi, 91, f2, g2,... approxi- 
mates f too, so the sequence I(f,), Z(g1), I(f2), [(g2),.-- has a limit in L?. But 
then all subsequences of the latter sequence, in particular, I(f,), I(f2),... and 
I(g:), I(g2),... have the same limit, which we denote by J(f). We have shown 
that 


IA 


A 


Jim |II(f) — 1(Fa)llc2 = 9, 


i.e. (7.5) holds for any sequence fj, fo,... of random step processes approxi- 
mating f. 
Finally, by Proposition 7.1 


(fr )ilp2 a lfnll are 


for each n, since the f, are random step processes. By taking the limit as 
n — oo we obtain 


WA c2 = UF lle - 
But this is equality (7.6). 0 


1 To be precise, the norms are defined on classes of functions, respectively, from M? 
and L“ determined by the relation of equality a.s. However, we shall follow the 
custom of identifying such classes with any of their members. 
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Exercise 7.4 


Show that for any f,g € M? 


Bis) = B( [ sateat), 


Hint Write the left-hand side in terms of F (\Z(f) + I(g)|") and # (\I(f) — I(g9)|*), 
the right-hand side in terms of E Cis [f(t) + g(t)|? dt) and & (fo [f(t) — g(t)’ dt) 
and then use (7.6). 


Having defined the It6 stochastic integral from 0 to oo, we are now in a 
position to consider stochastic integrals over any finite time interval [0, 7). 


Definition 7.5 


For any T > 0 we shall denote by M2 the space of all stochastic processes 
f(t),t > 0 such that 


loot) f eS M? 
The It6 stochastic integral (from 0 to T) of f € M2 is defined by 
Ir(f) =I (lp,ryf) - (7.7) 
We shall also write = 
[ soawe 
0 


in place of I7(f). 


Exercise 7.5 


Show that each random step process f € Méep belongs to M? for any t > 0 
and 


t 
LA = [ fs)aw(s 
sa martingale. 


Hint The stochastic integral of a random step process f is given by the sum (7.3). 
What is the conditional expectation of the jth term of this sum given Ff; if s < t;? 
What is it when s > t;? 


The processes for which the stochastic integral exists have been defined 
1S those that can be approximated by random step processes. However, it is 
10t always easy to check whether or not such an approximation exists. For 
yractical purposes it is important to have a straightforward sufficient condition 
‘ar a nrocess ta have a stochastic integral. In calculus there is a well-known 
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_ result of this kind: the Riemann integral exists for any continuous function. 
Here is a theorem of this kind for the It6 integral. 


Theorem 7.1 


Let f(t),t > 0 be a stochastic process with a.s. continuous paths adapted to 
the filtration F;. Then 


1) f € M?, ie. the Itd integral J(f) exists, whenever 
[e @) 
E (/ (oP at) < 00; (7.8) 
0 


2) f € M2, ie. the Ité integral I7(f) exists, whenever 


ad 2 
E (/ | F(t)| i) < 00. (7.9) 


Proof 


1) Suppose that f(t),t > 0 is an adapted process with a.s. continuous paths. If 
(7.8) holds, then 


k 
= ket< = 1,2,...,n?- 
fa(t) f(s) ds &<t< == fork =1,2,...,n*—1, (7.10) 


otherwise, 


is a sequence of random step processes in Mixep: Observe that for any k = 


DD ike 
k+l k 2 k 
| a(t)? dt =n j flat < | ‘FP dt as. (7.11) 
& ulasat cada § 
by Jensen’s inequality. We claim that 
Oo 
lim If(t) — fn(t)|?dt=0 as. 
12— CO 0 


This will imply that 
lim EB ( | if (t) fu(t) dt =0 


by the dominated convergence theorem and condition (7.8) because 
Ooo e.¢) 
[ we-sora s2f (oP + lal) at 
0 
e.@) 
< a[ |f(t)|? de. 
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The last inequality follows, since 


Pimoras [ior d as 
0 0 


for any 7, by taking the sum from k = 0 to oo in (7.11). 
To verify the claim observe that 


fore) N , fore) ‘ 
/ F(t) — fa(t)|? dt / If (t) — f(t) [2 dt + : F(t) — fin(t)|? dt 
0 @) N 


< | [f= falt)Pat +2 [ (so? + ator) a 


N 


IA 


N 8a : 
| F(t) — fold dt+4 | [f(t)| dt as. 


N-1 


The last inequality holds because 
oO fore) fore) 5 
/ inte)? at < f lf (t)|? ats | If(t)|° dt as. 
N N-i N-1 


for any n and JN, by taking the sum from k = mN to oo in (7.11). The claim 
follows because 


lim f(t)’ dt =0 as. 
N- oo N-1 


by (7.8) and 
N 


lim FOa=fi)l\ dO. xs: 


n—- Co 0 
for any fixed N by the continuity of paths of f. 

The above means that the sequence f), fo,... € Mien approximates f in 
the sense of Definition 7.3, so f € M?. 

2) If f satisfies (7.9) for some T > 0, then ljo,7)f satisfies (7.8). Since f is 
adapted and has a.s. continuous paths, 1j9,7)f is also adapted and its paths are 
a.Ss. continuous, except perhaps at T’. But the lack of continuity at the single 
point T does not affect the argument in 1), so lio7mf € M 2. This in turn 
implies that f € M2, completing the proof. 0 


Exercise 7.6 


Show that the Wiener process W(t) belongs to M2 for each T > 0. 


Hint Apply part 2) of Theorem 7.1. 


Exercise 7.7 


OW. aba TAT SAND LAlianca ta AL2 far anch TN 
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Hint Once again, apply part 2) of Theorem 7.1. 


The next theorem. which we shall state without proof, provides a character- 
ization of M? and M3, i.e. a necessary and sufficient condition for a stochastic 
process f to belong to M? or M2. It involves the notion of a progressively 
measurable process. 


Definition 7.6 


A stochastic process f(t),t > 0 is called progressively measurable if for any 
t>0 
(s,w) ++ f(s,w) 


is a measurable function from [0,t] x 2 with the o-field B[0,t]xF to R. Here 
B{0,t]xF is the product o-field on [0,t] x 2, that is, the smallest o-field con- 
taining all sets of the form A x B, where A C (0,t] is a Borel set and B € F. 


Theorem 7.2 


1) The space M? consists of all progressively measurable stochastic processes 


f(t),t > 0 such that 
(e.@) 
E ¢ f(t) < 00. 
0 
2) The space M2 consists of all progressively measurable stochastic processes 


f(t),¢ > 0 such that 
2 at < 0O 
E : 


7.2 Examples 


According to Exercise 7.6, the Wiener process W(t) belongs to M?# for any 
T > 0. Therefore the stochastic integral in the next exercise exists. 


Exercise 7.8 


Verify the equality 
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by computing the stochastic integral from the definition, that is, by approxi- 
mating the integrand by random step functions. 


Hint It is convenient to use a partition of the interval [0,7] into n equal parts. The 
limit of the sums approximating the integral has been found in Exercise 7.1. 
Exercise 7.9 

Verify the equality 


T T 
[ caw =rwiry- [ W (t) dt, 
0 0 


by computing the stochastic integral from the definition. (The integral on the 
right-hand side is understood as a Riemann integral defined pathwise, i.e. sep- 
arately for each w € J2.) 


Hint You may want to use the same partition of [0,7] into n equal parts as in 
Solution 7.8. The sums approximating the stochastic integral can be transformed 
with the aid of the identity 


c(b — a) = (db — ca) — b(d—c). 


Exercise 7.10 
Show that W(t)? belongs to M2 for each T > 0 and verify the equality 


T T 
1 
[ W(t)aWit) = -W(T) - i; W(t) dt, 
0 0 
where the integral on the right-hand side is a Riemann integral. 


Hint As in the exercises above, it is convenient to use the partition of [0,7] into n 
equal parts. The identity 


a? (b — a) = 5 (6° 0°) ~a(b—a)? ~ 5 (b- a) 


can be applied to transform the sums approximating the stochastic integral. You may 
also need the following identity: 


(a? —b?)* = (a—b)' +.4(a—b)?b+4 (a —b)? 07. 


7.3 Properties of the Stochastic Integral 


The basic properties of the It6 integral are summarized in the theorem below. 
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Theorem 7.3 


The following properties hold for any f,g € M?, any a,@ € R, and any 0 < 
s<t: 


1) linearity 


i (af (r) + 8g(r)) dW(r) = 0 / f(r) dW(r) +8 / g(r) dW (r); 
0 O O 


e( ) =2(f isnPar). 


3) martingale property 


2) isometry 


| “ f(r) aw(r) 


E ([ 70) dW (r) F.| = i f(r) dW(r). 


Proof 


1) If f and g belong to M/, then 114) f and 1[0,4)9 belong to M”, so there are 
sequences ff, fo,... and gi,g2,... in Ma approximating lig) f and lio 4)g. It 
follows that 1104) (af + Bg) can be approximated by af; + 6g1,af2 + Bgo,.... 
By Exercise 7.3 

I (afn + Bgn) = al (fn) + BI(gn) 
for each n. Taking the L? limit on both sides of this equality as n — oo, we 
obtain 
I (1jo,t) (af + Bg)) = aT (loz) f) + BI(1o,2)9); 
which proves 1). 

2) This follows by approximating 1(9,4) f by random step processes in Miep 
and using Proposition 7.1. 

3) If f belongs to M?, then 194) f belongs to M?. Let f;, f2,... be a se- 


quence of processes in Miep approximating lj 4) f. By Exercise 7.5 
E (I (1jot) fn) IFs) = I (1Jo,s) fn) (7.12) 


for each n. By taking the L? limit of both sides of this equality as n > 00, we 
shall show that 

E (I (1po,t)f) Fs) = J (los) f) 
which is what needs to be proved. Indeed, observe that 119.5) f1, l[o,s) fo,.-. is a 


sequence in M3,,, approximating 1jo,s)f, so 


I (1jo,s) fn) >I (1j0,s)f) in L? as n > oo. 
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similarly, 1/0.) fi ljo,1)f2,--- is also a sequence in Mess approximating ljo,1) f, 


vhich implies that 
I (loa fn) 21 (poz)f) in L* asn > ov. 
[he lemma below implies that 
E (1 (lpt)fn) |Fs) 2 E (7 (pot) f) |Fs) in L? asn > 0, 


completing the proof. 0 


-emma 7.1 


f € and 1, &,... are square integrable random variables such that €, — € in 


"? as n — oo, then 


E(€,|G) > E(€|G) in L? asn - oo 


or any o-field G on §2 contained in F. 


>roof 


3y Jensen’s inequality, see Theorem 2.2, 
IE (EnlG) - B (E19)? = |B (& ~ 19)? < E(l&m -€P|9), 
vhich implies that 


E (IE (£19) - E(éI9)I) 


IA 


E(E (lf - €"|9)) 


E (| -€’) +0 


ison—-oo. U 


In the next theorem we consider the stochastic integral ie f(s)dW(s) asa 
unction of the upper integration limit t. Similarly as for the Riemann integral, 
t is natural to ask if this is a continuous function of t. The answer to this 
tuestion involves the notion of a modification of a stochastic process. 


definition 7.7 


set €(t) and C(t) be stochastic processes defined for t € T, where T C R. We 
ay that the processes are modifications (or versions) of one another if 


PJS¢(*) —f(*1 —1 far all te T (7 12) 


watered 
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Remark 7.1 
If T Cc R is a countable set, then (7.13) is equivalent to the condition 
P {€(t) = C(t) for all te T} = 1. 


However, this is not necessarily so if T is uncountable. 


The following result is stated without proof. 


Theorem 7.4 


Let f(s) be a process belonging to M? and let 


ae) = [ssa 


for every t > 0. Then there exists an adapted modification ¢(t) of €(t) with a.s. 
continuous paths. This modification is unique up to equality a.s. 


From now on we shall always identify ie f(s) dW(s) with the adapted mod- 
ification having a.s. continuous paths. This convention works beautifully to- 
gether with Theorem 7.1 whenever there is a need to show that a stochastic 
integral can be used as the integrand of another stochastic integral, i.e. belongs 
to M2 for T > 0. This is illustrated by the next exercise. 


Exercise 7.11 
Show that ; 
ee) = | W(s)av(s 


belongs to M4 for any T > 0. 


Hint By Theorem 7.4 €(t) can be identified with an adapted modification having a.s. 
continuous trajectories. Because of this, it suffices to verify that &(t) satisfies condition 
(7.9) of Theorem 7.1. 


7.4 Stochastic Differential and Ito Formula 


Any continuously differentiable function z(t) such that r(0) = O satisfies the 
formulae 


ig 
a(T)? = 2 | z(t) dx(t), 
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T 
z(T)>? = 3 | a(t)? dz(t), 


where dx(t) can simply be understood as a shorthand notation for z’(t) dt, the 
integrals on the right-hand side being Riemann integrals. Similar formulae have 
been obtained in Exercises 7.8 and 7.10 for the Wiener process: 


= 
3 
I 


[ dt + of W(t) dW (t), 


= 
3 
I 


T T 
2 
3 | W(t) a+ [ W(t)2dW (t). 


Here the stochastic integrals resemble the corresponding expressions for a 
smooth function z(t), but there are also the intriguing terms (Ka dt and 


3 i W (t) dt. The formulae for W(T)? and W(T)? are examples of the much 
more general J/t6 formula, a crucial tool for transforming and computing 
stochastic integrals. Terms such as i dt and 3 iid W (t) dt, which have no ana- 
logues in the classical calculus of smooth functions, are a feature inherent in 
the It6 formula and referred to as the J/t6 correction. The class of processes 
appearing in the Ito formula is defined as follows. 


Definition 7.8 


A stochastic process €(t),¢t > 0 is called an J/t6 process if it has a.s. continuous 
oaths and can be represented as 


T T 
€(T) = €(0) + f a(t) a+ | b(t)dW(t) as., (7.14) 


where b(t) is a process belonging to M2 for all T > 0 and a(t) is a process 
idapted to the filtration 7; such that 


T 
i: la(t)|dt<oo as. (7.15) 
0 


or all T > 0. The class of all adapted processes a(t) satisfying (7.15) for some 
r > 0 will be denoted by L4. 
For an It6 process € it is customary to write (7.14) as 
d&(t) = a(t) dt + b(t) dW(t) (7.16) 


ind to call df(t) the stochastic differential of €(t). This is known as the It6 
lifferential notation. It should be emphasized that the stochastic differential 


ae na wall-definead mathamatical maaninea an ite awn and achaiwldA aleracra ha 


7. Ité Stochastic Calculus 195 


understood in the context of the rigorous equation (7.14). The Ito differential 
notation is an efficient way of writing this equation, rather than an attempt to 
give a precise mathematical meaning to the stochastic differential. 


Example 7.1 


The Wiener process VV (t) satisfies 


T 
W(T) = i dW (t). 


(The right-hand side is the stochastic integral I(f) of the random step process 
f = l0,7)-) This is an equation of the form (7.14) with a(t) = 0 and b(t) = 1, 
which belong, respectively, to Cj, and M? for any T > 0. It follows that the 
Wiener process is an [t6 process. 


Example 7.2 


Every process of the form 


where a(t) is a process belonging to CL}. for any T > 0, is an It6 process. In 
particular, every deterministic process of this form, where a(t) is a deterministic 
integrable function, is an It6 process. 


Example 7.3 


Since a(t) = 1 and b(t) = 2W(t) belong, respectively, to the classes £4. and 
M+? for each T > 0, 


W(T)? = rE dt + of W (t) dW(t) 


is an It6 process; see Exercise 7.8. The last equation can also be written as 
d(W(t)’) = dt+2W/(t) dW(t), 


providing a formula for the stochastic differential d(W(t)?) of W(t)?. 


Exercise 7.12 


Show that W(t)? is an It6 process and find a formula for the stochastic differ- 
ential d(W(t)°). 
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Hint Refer to Exercise 7.10. 


Exercise 7.13 

Show that tW(t) is an It6 process and find a formula for the stochastic differ- 
ential d(tW/(t)). 

Hint Use Exercise 7.9. 


The above examples and exercises are particular cases of an extremely im- 
portant general formula for transforming stochastic differentials established by 
It6. To begin with, we shall state and prove a simplified version of the formula, 
followed by the general theorem. The proof of the simplified version captures 
the essential ingredients of the somewhat tedious general argument, which will 
be omitted. In fact, many of the essential ingredients of the proof are already 
present in the examples and exercises considered above. 


Theorem 7.5 (It6 formula, simplified version) 


Suppose that F(t,z) is a real-valued function with continuous partial deriva- 
tives Fy(t,z), Fi(t,z) and Fi (t,x) for all t > 0 and z € R. We also assume 
that the process F/(t,W(t)) belongs to M4 for all T > 0. Then F(t, W(t)) is 


an Ito process such that 
T 
F(r.W(T)) - FO,W()) = [ (WO) + ZF(LW EO) ) a 
T 
+/ F(t, W(t))dW(t) as. (7.17) 
0 


In differential notation this formula can be written as 


dF(t,W(t)) = (Faw) + 5 Fee(t W(t) ) dt + Fi(t, W(t)) dW (t). (7.18) 


Remark 7.2 
Compare the latter with the chain rule 
dF (t,x(t)) = Fy(t,2(t)) dt + F,(t, z(t)) dz(t). 


for a smooth function x(t), where dz(t) is understood as a shorthand notation 
for x'(t)dt. The additional term $FY!,(t,W/(t)) dt in (7.18) is called the [té 


correction. 
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Proof 


First we shall prove the Ito formula under the assumption that F' and the 
partial derivatives F, and F’’. are bounded by some C' > 0. 

Consider a partition 0 = tj < t? < --- < t® = T, where t? = ae of 
[0,7] into n equal parts. We shall denote the increments W(t?,,) — W(t?) 
by A?W and t?,., — t? by A?t. We shall also write W?" instead of W(t?) for 
brevity. According to the Taylor formula, there is a point W;" in each interval 
[W (t?), W(t?,,)] and a point t? in each interval [t?, t?,,] such that 


n—1l 


F(T,W(T)) — F(0,W(0)) = d> (F(tha, Wha) — FP, WP) 
i=0 
n—-1l n—l 
= SS (F(t21,Wii) — F?, Wi) + Ly (F(t, Wii) — F(t?,W?)) 
1=0 1=0 
n—l1 = 1 
in n n n n n n n 2 
= re see ray 4D t? WP)At PW+ 5D Bes (t?, W") (APW) 
n-1 n—-1l 
= SoM (, Wa )Art + = 5 Fl (EP, WP) Attt+ So Fi(te, Wr) Atw 
i=0 i=0 


_ y Ftc, WP) ((Apw)? — det) 


1 7 

mS |Fes(?, Wi) — Feo(t?,W)| (APW)?. 
a 0 

We shall deal separately with each sum in the last expression, splitting the 

proof into several steps. 


Step 1. We claim that 


T 
n n t 
un, Fi(t?, Wi) Aj ee Fy(t,W(t))dt as. 


0 


This is because the — of W(t) are a.s. continuous, and F/(t, z) is continuous 
as a function of two variables by assumption. Indeed, every continuous path 
of the Wiener process is bounded on [0,7], i.e. there is an M > 0, which may 
depend on the path, such that 


\|W(t)| <M for all t € (0,7). 


As a continuous function, Fy(t,z) is uniformly continuous on the compact set 
{0,T] x [—M, M] and W is uniformly continuous on [0,7]. It follows that 


lim sup |Fy(é?,W2.1) -Fi(t,W(t))|=0 as., 
M—+0O 432 
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where the supremum is taken over all i=0,...,n—1 and t € [t?,t?,,]. By the 
definition of the Riemann integral this proves the claim. 
Step 2. This is very similar to Step 1. By continuity 


lim sup [FY (8, WP) — Fi, (t, W(t))|=0 as., 
TlL— CO i,t 


where the supremum is taken over alli = 0,...,n —l andt € [t?, ¢?,,]. By the 
definition of the Riemann integral 


lim 2 Fal t?, WwW) yare= [ Fi.(t,W(t))dt as. 
0 
Step 3. We shall verify that 
n— i & 
Jim, Fee weyaw = | F'(t,W(t))dW(t) in L?. 


If Fi (t,x) is bounded by C > 0, then f(t) = Fi(t,W(t)) belongs to M#, by 
Theorem 7.1, and the sequence of random step processes 


fn = D_ Fa (t?, WP) 1pm en.) © Mitep 
approximates f. Indeed, by continuity 
Jim lfn(t) — f(t)|? =0 foranyte [0,7], as. 


Because |f,(t) — f(t)|? < 4C?, it follows that 


lim, | lfn(t) — f(t)|?dt=0 as. 


by Lebesgue’s dominated convergence theorem. But i | fn(t) — f(t)|? dt < 


4T C7, so 
T 
lim £ (/ |fn(t) - 0a =0 
noo 0 


again by Lebesgue’s dominated convergence theorem. This shows that fy ap- 
proximates f, which in turn implies that I(fn,) tends to I(f) in L?, concluding 
Step 3. 

Step 4. If Fy, is bounded by C > 0, then 


n—1 
Jim SO (8, WP) ((apw)? = Art) =0 in L?, 


1=0 
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since 


n—1 2 


D Fes(t?, Wh) ((APW)? — Ape) 


7=0 


n—1 


E 


Fets(e, WE) ((apW)? ~ Ape) | 


n—l : 
= Ele (et, WeyP Baw) - are 


i=0 


n—-1 n—1 
<C’S(E \(arwy? — Art| = 20” S_ (Art)? 


i=0 1=0 


n-l 
T? T? 
= 20? 5° — =20?— +0 as nN —> 00. 
ar Tr 


The first equality above holds because for any 1 < j 


E [Fe (en, Wy") ((APw)? — Ast) Fi, (, Wp) ((42w)? - At) | 


= B [Fi tn, WP) ((ApW)? - Ate) Fa.(3,WP)] B[(AzW)? - ape 
= 0. 
This is because the expressions in the last two square brackets are independent 


and the last expectation is equal to zero. 
Step 5. By a similar continuity argument as in Steps 1 and 2 


lim: sup Fy (t?,W?) — Fi (t?,W?))=0 as., 
where the supremum is taken over alli = 0,1,...,2—1. Since Ss. (Arw)? > 


T in L? as n > ov, there is a subsequence n, < ng <... such that 
nte—l 
So (APWY aT as. 
1=0 

as k > oo. It follows that 


npe—l 


» (FL Ae ly tea ce Caer w*)) (A™ Ww)? 
1=0 
me ne—1 
< sup [FY (tP*, Witt) — Fr (t*,W24)| So (AP*W)? 30 as. 
a ar 


as k > oo. 
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In those steps above where L? convergence was obtained, we also have 
convergence a.s. by taking a subsequence. This proves the Ito formula (7.17) 
under the assumption that the partial derivatives F{(t,z) and FY',(t,2) are 
bounded. To complete the proof we need to remove this assumption. Let F(t, x) 
be an arbitrary function satisfying the conditions of Theorem 7.5. For each 
positive integer n take a smooth function y, from R to (0, 1] such that y,(z) = 
1 for any xz € [—n,n] and y, (x) = 0 for any z ¢ [—n —1,n 4+ 1]. Then 


Fr(t, ©) = ~n(x) F(t, x) 


also satisfies the conditions of Theorem 7.5 and has bounded partial derivatives 
(F,,)),.(t,z) and (F,,)i..(t, z) for each n. Therefore, by the first part of the proof 


F(T, W(T)) ~~ F,, (0, W(0)) 
T 1 T 
= [ (ewe) + 5iZwO)) a+ [ Frew) aE. 


Consider the expanding sequence of events 


Since F(t,z) = F,(t.z) for every t € [0,7] and x € [—n,n], it follows that 
(7.17) holds on A,. It. remains to show that 


Jim PAs) = 1 


to prove that (7.17) holds a.s. But the latter is true because of Doob’s maximal 
[? inequality, Theorem 6.7, which implies that 


n’ (1— P(An)) = “P| sup |W(t)| > | 
t€[0,T] 


completing the proof. UO 


Example 7.4 


For F(t, x) = 2? we have Fy (t, rz) = 0, Fi(t,xz) = 22 and F",(t,x) = 2. The Ité 
‘ormula gives 
d(W(t)?) = dt + 2W(t) dW(t), 


which is the same eaualitv as in Exercise 7.8. 
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Example 7.5 
For F(t,r) = z° we have Fy (t,z) = 0, Fi(t,r) = 32? and Fi'.(t,r) = 62. By 
the It6 formula we obtain the same equality 


d(W(t)?) = 3W(t) dt + 3W(t)* dW(t) 


as in Exercise 7.10. 


Exercise 7.14 (exponential martingale) 


L 


Show that the exponential martingale X(t) = e”e—2 is an Itd process and 


verify that it satisfies the equation 


dX(t) = X(t) dW(t). 


Hint Use the Ité formula with F(t, zr) = eee. 


As compared with the simplified version just proved, in the general Ité 
formula below W(t) will be replaced by an arbitrary It6 process €(t) such that 


dé(t) = a(t) dt + b(t) dW(t), (7.19) 


where a belongs to L} and b to M? for all t > 0. In the general case the proof 
will be omitted. 


Theorem 7.6 (It6 formula, general case) 


Let £(t) be an It6 process as above. Suppose that F(t, x) is a real-valued func- 
tion with continuous partial derivatives F/(t,x), Fy (t,2) and Fy/,(t, x) for all 
t > 0 and z € R. We also assume that the process b(t) F(t, €(t)) belongs to 
M®# for all T > 0. Then F(t, €(t)) is an It6 process such that 


dF(t,€(0) = (FUCEE()) + Fults €(t)) alt) + SFee(t,€(0) (0? ) at 


+ F(t, €(t)) b(t) dW(t). (7.20) 


A convenient way to remember the It6 formula is to write down the Taylor 
expansion for F(t,xz) up to the terms with partial derivatives of order two, 
substituting €(t) for « and the expression on the right-hand side of (7.19) for 
dé(t), and using the so-called Jté multiplication table 


dt dt = 0, dt dW(t) = 0, 
dW(t)dt=0, dW(t)dW(t) = dt. 
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his informal procedure gives 


1 1 
IF = Fidt+ F.d€+ 5Fudtdt + Fy dtdé + 5 Fy, dé dé 


= F,/ dt+ F, (adt+ bdW) 
1 


+5Fydt dt + Fy, dt (adt + bdW) + =F, (adt + bdW) (adt + bdW) 


= Fidt+ F! (adt+bdW) + SFiy dt 


] 
(F + Flat+ Fist?) dt + Fb dW, 


hich is the expression in (7.20). Here we have omitted the arguments (t, €(t)) 
id, respectively, (t) in all functions for brevity. 


xercise 7.15 


pplying the It6 formula to F(t,z) = x2", show that 


dW(t)" = mn yy dt + nW(t)"-} dW (t) (7.21) 


aint This is a direct application of the It6 formula, but be careful with the assump- 
ons, in particular make sure that nW(t)"~! belongs to M?# for all T > 0. 


xercise 7.16 (Ornstein—Uhlenbeck process) 


uppose that a > 0 and o € R are fixed. Define Y(t),t > 0 to be an adapted 
iodification of the Ité integral 


t 
Ya) = eet | e*S dW(s) 


ith a.s. continuous paths. Show that Y(t) satisfies 


dy (t) = —aY(t) dt+ odWi(t) 


int Y(t) = F(t, €(t)) with €(t) = of, e°* dW(s) and F(t,r) =e °'s. 


".5 Stochastic Differential Equations 


‘his section will be devoted to stochastic differential equations of the form 


d€(t) = f (E(t) dt + g(€(t)) dW (t). 
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Solutions will be sought in the class of It6 processes €(t) with a.s. continuous 
paths. As in the theory of ordinary differential equations, we need to specify 
an initial condition 


€(0) = £0. 


Here 9 can be a fixed real number or, in general, a random variable. Being an 
It6 process, €(t) must. be adapted to the filtration F; of W(t), so & must be 
Fo-measurable. 


Example 7.6 
The stochastic differential equation 
dX (t) = X(t) dW(t) (7.22) 


was used as a motivation for developing It6 stochastic calculus at the beginning 
of the present chapter. In Exercise 7.14 it was verified that the exponential 


martingale 
Nit) = eV (te-2 


satisfies (7.22). It also satisfies the initial condition X(0) = 1. This is an ex- 
ample of a linear stochastic differential equation. For the solution of a general 
equation of this type with an arbitrary initial condition, see Exercise 7.20. 


Example 7.7 


In Exercise 7.16 it was shown that the Ornstein—Uhlenbeck process 
t 
Y(@):= venet | e** dW (s) 
0 
satisfies the stochastic differential equation 
dY (t) = —aY(t) dt + o dW (t) 


with initial condition Y(0) = 0. This is an example of an inhomogeneous lin- 
ear stochastic differential equation. See Exercise 7.17 for a solution with an 
arbitrary initial condition. 


Definition 7.9 


An It6 process €(t), t > 0 is called a solution of the initial value problem 


de(t) = f(€(t)) dt + g(E(t)) dW(t), 
€(0) = &0 
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if &) is an Fo-measurable random variable, the processes f(€(t)) and g(€(t)) 
belong, respectively, to C4, and M?, and 


ar) =c+ | f(é wars g(E(t)) dW(t) as. (7.23) 


for all TJ’ > 0. 


Remark 7.3 


In view of this definition, the notion of a stochastic differential equation is 
a fiction. In fact, only stochastic integral equations of the form (7.23) have a 
rigorous mathematical meaning. However, it proves convenient to use stochastic 
differentials informally and talk of stochastic differential equations to draw on 
the analogy with ordinary differential equations. This analogy will be employed 
to solve some stochastic differential equations later on in this section. 


The existence and uniqueness theorem below resembles that in the theory 
of ordinary differential equations, where it is also crucial for the right-hand side 
of the equation to be Lipschitz continuous as a function of the solution. 


Theorem 7.7 


Suppose that f and g are Lipschitz continuous functions from R to R, i.e. there 
is a constant C' > 0 such that for any z,y € R 


f(z) -—fly)| < Clz— yl, 
lg(z) —g(y)| < Clr —yl. 


Moreover, let £9 be an Fo-measurable square integrable random variable. Then 
the initial value problem 


d€(t) = f(€(t)) dt + g(E(t)) dW(t), (7.24) 
€(0) = €0 (7.25) 


has a solution €(t),¢ > 0 in the class of It6 processes. The solution is unique in 
the sense that if n(t),t > 0 is another It6 process satisfying (7.24) and (7.25), 
then the two processes are identical a.s., that is, 


P {€(t) = n(t) for all t > 0} = 1. 


7. Ité Stochastic Calculus 205 


Proof (outline) 


Let us fix T > 0. We are looking for a process £ € M# such that 


ator | rE F(€ pyar + | g(E(t))dW(t) as. (7.26) 


for all s € [0,7]. Once we have shown that such a € € M2, exists, to obtain 
a solution to the stochastic differential equation (7.24) with initial condition 
(7.25) it suffices to take a modification of € with a.s. continuous paths, which 
exists by Theorem 7.4. 

To show that a solution to the stochastic integral equation (7.26) exists we 
shall employ the Banach fixed point theorem in M# with the norm 


T 
lel = 2 / eM e(t) 2 at, (7.27) 


which turns M# into a complete normed vector space. The number \ > 0 
should be chosen large enough, see below. To apply the fixed point theorem 
define  : M2, > M2 by 


= tor fH F(é pats g(E(t)) dW (t) (7.28) 


for any € € M2 and s € [0,T]. We claim that © is a strict contraction, i.e. 


P(E) — Pla < allE — Cla (7.29) 


for some a < 1 and all £,¢ € M2. Then, by the Banach theorem, & has a 
unique fixed point € = ®(€). This is the desired solution to (7.26). 

It remains to verify that @ is indeed a strict contraction. It suffices to show 
that the two maps #; and #2, where 


=[ rewa, ee = [ aware, 
J0 0 


are strict contractions with contracting constants a, and a2 such that a; +a < 
1. For ¢; this follows from the Lipschitz continuity of f. For 2 we need to use 
the Lipschitz continuity of g and the isometry property of the It6 integral. Let 
us mention just one essential step in the latter case. For any £,¢ € M2 


T 8 
E / es [ (g(€(t)) — 9(C(t))] aw) 


T 8 
=E [ eo ; lo(E(t)) — g(C(t))[? deds 


oe fe [lee - C(t)|* dt ds 


2 
ds 


IIP2(€) — S2()Ih 


lA 
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T T 
C*E —AXs At d —xXt x t 2 dt 
i < : ‘Je le(t) — c(t) 
Co 25 ee, C" 2 
< Sef eet) cP a= Sle oik, 


ince (i eet ds = $(1— e779) < 1. Here C is the Lipschitz constant 
f g. If \ > C*/e, then 2 is a strict contraction with contracting constant < e. 

There remain some technical points to be settled, but the main idea of the 
roof is shown above. O 


=xercise 7.17 
‘ind a solution of the stochastic differential equation 
dX (t) = —aX(t) dt +adW(t) 


ith initial condition X(0) = zo, where Zo is an arbitrary real number. Show 
hat the solution is unique. 


lint Use the substitution Y(t) = e*’ X(t). 
A linear stochastic differential equation has the general form 
dX (t) = aX(t) dt + bX (t)dW(é), (7.30) 


7here a and 0 are real numbers. In particular, for a = 0 and b = 1 we obtain the 
tochastic differential equation dX (t) = X(t) dW(t) in Example 7.6. The solu- 
ion to the initial value problem for any linear stochastic differential equation 
an be found by exploiting the analogy with ordinary differential equations, as 
resented in the exercises below. 


-xercise 7.18 


uppose that w(t), t > 0 is a deterministic real-valued function of class C+ such 
hat w(0) = 0. Solve the ordinary differential equation 


dx(t) = ax(t) dt + bx(t) dw(t), (7.31) 
rith initial condition z2(0) = zo to find that 
a(t) = xpettt oul), (7.32) 


We write dw(t) in place of w'(t) dt to emphasize the analogy with stochastic 
ifferential equations.) 
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Hint The variables can be separated: 
dz(t) 
x(t) 


By analogy with the deterministic solution (7.32), let us consider a process 
defined by 


= (a+ bw'(t)) dt. 


X(t) Saxe © (7.33) 


for any t > 0, where IV(t) is a Wiener process. 


Exercise 7.19 


Show that X(t) defined by (7.33) is a solution of the linear stochastic differential 
equation ; 
b 
dX (t) = (« + _ X(t) dt + bX (t) dW (t), (7.34) 
with initial condition X (0) = Xo. 
Hint Use the It6 formula with F(t, xz) = e®'t®*. 


Exercise 7.20 


Show that the linear stochastic differential equation 
dX (t) = aX (t) dt + bX (t) dW(t) 
with initial condition X (0) = Xo has a unique solution given by 
X(t) = Xpelan te etow le), 
Hint Apply the result of Exercise 7.19 with suitably redefined constants. 


Having solved the general linear stochastic differential equation (7.30), let 
us consider an example of a non-linear stochastic differential equation. Once 
again, we begin with a deterministic problem. 


Exercise 7.21 


Suppose that w(t), t > 0 is a deterministic real-valued function of class C! such 
that w(0) = 0. Solve the ordinary differential equation 


dx(t) = f1+ z(t)? dt+ 1+ z(t)? dw(t) 
with initial condition 2(0) = zo. 


Hint The variables in this differential equation can be separated. 
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ercise 7.22 
ow that the process defined by 
X(t) = sinh(C + t+ W(t)), 


ere W(t) is a Wiener process and C = sinh~! Xo, is a solution of the stochas- 
differential equation 


= (vi + X(t)? + 5x) dt + ( 1+ X(t?) dW (t) 
th initial condition X(0) = Xo. 


nt Use the It6é formula with F(t, 2) = sinh(t + 2). 


We shall conclude this chapter with an example of a stochastic differen- 
] equation which does not satisfy the assumptions of Theorem 7.7. It turns 
t that the solution may fail to exist for all times t > 0. This is a familiar 
enomenon in ordinary differential equations. However, stochastic differential 
uations add a new effect, which does not even make sense in the deterministic 
se: the maximum time of existence of the solution, called the explosion time 
1y be a (non-constant) random variable, in fact a stopping time. 


cample 7.8 
msider the stochastic differential equation 
dX (t) = X(t)?dt + X(t)?dW(t). 
en tae 1 
~ 1-Wi(t) 


a solution, which can be verified, at least formally, by using the It6 formula 
th F(t, 2) = j = (t) exists only up to the first hitting time 


7 =inf {t >0:W(t) =1} 


iis is the explosion time of X(t). Observe that 
: na 
Bee 


rictly speaking, the Itd formula stated in Theorem 7.6 does not cover this 
se, since F(t,#) = ;+ i = 1. Definition 7.9 does not 
iply either, as it requires the solution X(t) to be defined for all t > 0. Suitable 
tensions of the It6 formula and the definition of a solution are required to 
udy stochastic differential equations involving explosions. However, to prevent 
l i gas of this book, we -have to refer the interested reader to a further 
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7.6 Solutions 


Solution 7.1 


Using the first identity in the hint we obtain 


We) (We) — WED) = 5 (Wa)? - WED?) 


j=0 
jee 9 
9 (W(t? 1) — W (ti; )) 
j=0 
= dwery- 2 (wie) - WEN)? 
j=0 


By Exercise 6.29 the limit is 
te 1 1 
: n\\ _ 2 
Jim, oe W (th) (W(th41) — W(t})) = SW(T)? — ST. 
J= 
Similarly, the second identity in the hint enables us to write 


Sweets) (Wes. — WEEN) = BS 9 (W (8.1)? - WY?) 
j=0 


(W(th41) - wit?))° 


209 


= =W(T)’+ yw (8,1) — WER)”. 


It follows that 


n—-1 
; n n n 1 1 
im, DoW (thes) (W(G2) — WED) = SWE) + 5 
J= 


Solution 7.2 


For any random step processes f,g € Mee, there is a partition 0 = to < t) < 


++ <t, such that for any t > 0 


n—-1 n— 
t) = 3 15 lies t541) (¢) and g(t) = > Ci lte;.tj41) (2), 
j=0 j=0 


where 7; and ¢; are square integrable F;,-measurable random variables for each 
ee 0 n—1 (If the two partitions in the formulae for f and g happen to 
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ie different, then it is always possible to find a common refinement of the two 


‘artitions. ) 
As in the proof of Proposition 7.1, we denote the increment W (t;41) —W(t;) 


'V A,;Ww and tj41 > t; by AGT. Then 


n—-ln--l 


nj Ch A; WA,W 


I(f)I(g) 


(o>) 


i=0 k=: 
1 


= So mG lAiwl + >) Ge AWARW + > Gm AIWAW, 


0 j<k j<k 


mR. 
lI 


vhere, by independence, 
2 2 
E (nj; |A;W| ) = E (jG) E (|; ) = E (jj) Ajt 
nd 


E (nj Gr.4;W AW) 
EB (Cink A;W A.W) 


E (nj Chk A;W) FE (A.W) = 0 
E (Cjm A;W) E (A.W) = 0 


or any 7 < k. It follows that 


n—l 


E (I(f)I(g)) = >> E (nj) Ait. 


j=0 


“herefore, it suffices to show that 


E (fo f(t)g(t) it) = SB) A;t 


iut this is true because 


n-iln-l 


f(t)g9(t) = > Ss 715 Ce Ute; tj 41) (4) Lea tear) (t) 
j=0 k=0 
n—-1 


a 5651 (e,,t541) (4). 
j=0 
solution 7.3 


Ne shall use a partition 0 = tp < t) <-:+ < ty, such that 


n—l n—-1 


l= SS Nile, ,t541) and I= B Calte; ity4i)) 


7=0 


es msn RASA 
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where n; and ¢; are square integrable ¥;,-measurable random variables for each 
j = 0,1,...,n —1. (if the two partitions in the formulae for f and g happen 
to be different, then it is always possible to find a common refinement of the 
two partitions.) The increments W(t;41) — W(t,;) will be denoted by 4,;W for 
brevity. Then 


af + Bg = ¥ (ang +1965) Vests) 
and ~ 
I (af +g) = yon + BCj) AjW 
ae ” 
= a) AW +8) GAW 
= al(f) + BI(4). ~ 


Solution 7.4 


Consider the following scalar products in M? and L?: 


Gain = E( | * f()a(t) dt) and (nj Op. = B (nd) 


for any f,g € M? and n,¢ € L?. They can be expressed in terms of the 
corresponding norms defined in the proof of Proposition 7.2, 


l 1 
(f,9)ue2 = qllf + gle — gle = gllnee » 
1 1 
(1. Opa = zllnt Cllze — Zlln— Cz - 
Therefore Proposition 7.2 implies that 


(1(f),1(9))r2 =(F,9) ve » 


which is the same as the equality to be proved. 


Solution 7.5 


If f € Miep is a random step process, then so is ljo.4f € Miep C M? for any 
t > 0. This in turn implies that f € M? for any t > 0. 

We shall verify that J;(f) is a martingale with respect to the filtration F;. 
Let 0 < s < t and suppose that f € M3,, can be written in the form (7.2), 


where 


A=ta<ct <<: <th=s<tear << tm =t < tma <0: < tp. 
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ch a partition to,...,¢, can always be obtained by adding the points s and 
f necessary. We shall denote the increment W(tj;41) — W(t;) by 4;W as in 
2 proof of Proposition 7.1. Then 


m—1 
Lo f = Do ma Ues ts] 


j=0 


ee(f) = Ip f) = ya 


ich is adapted to F; and square iene and so integrable. It remains to 
npute 


m-1l1 
h(f)|Fs) = E (I(lpoyf)|\Fs) = >> E (ns A;W|Fs). 
7=—0 


j < k, then 7; and A;W are ¥,-measurable and 
E (nj; A;W|F.) = nj A;W. 
j > k, then ¥, C F;, and 


E(njA;W|Fs) = E(E (nj AjW|Fi;) |Fs) 
E (njE (A;W|Ft;) |Fs) 
E (nj|Fs) E (A;W) = 0, 


ce 7; is F;,-measurable and 4;W is independent of F;,. It follows that 


k-1 
E(h(f)|Fs) = S15 4;W = I(ljo,s)f) = Is(f). 


j=0 


lution 7.6 
definition, W(t) is adapted to the filtration F; and has a.s. continuous paths. 


yreover, 
T T 
E (/ |W (t)| a) = / E (|wie)| ) dt 


T 
-/ tdt < oo. 
0 


Theorem 7.1 it follows that the Wiener process W belongs to M2. 
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Solution 7.7 
Since W(t) is adapted to the filtration F;, so is W(t)?. Moreover, 


E ( [wear a) a [ =(wor) dt 


T 
= i 3t? dt < oo. 
6) 


Theorem 7.1 implies that W(t)? belongs to M2. 


Solution 7.8 
We fix JT’ > 0 and put 


f(t) = lor (t)W (2). 


[ woawe =| sane 


Take 0 = tf < t} <--- < t? = T, where t? = +, to be a partition of [0,7] 
into n equal parts, and put 


Then f € M? and 


n—l 


fn = > W (tj) Lier en,.)° 


i=0 


Then the sequence fi, fo,... € M2,,, approximates f, since 


tep 


E( [> ise) stat) = : [2 (|w(e) — Weer?) ae 


m=1 thy, 
= ae (t — t?) dt 


i=o ° 


By Exercise 7.1 


= SOW) (W(t) - WE) + SWZ)? - ST 
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_L? as n—- oo. We have found, therefore, that 


J 1 
0 


olution 7.9 


et f(t) = +t. Then ljo,7f belongs to MZ. 
',T'] into n equal parts as in Solution 7.8. 


fn = 3 t? Lien en, tn 


»proximates lig 7) f, since 


EB (| Lior) f(t) — fn(t)|” it) 


W(T)? — T. 


We shall use the same partition of 
The sequence 


Mitep 
- 2 
E ( [ 0 - mol a) 
0 
n—-1 biag 
S- jt — t?|° dt 
ix Vt? 
l n—-l T3 
3 n3 
t=1 
T3 
—~ +0 asn—-oo. 
3n? 


7ith the aid of the identity in the hint, we can write the stochastic integral of 


as 


He) = De (w (2,4) — W(t?) 


n—-1 


n—-1 


= D0 RW (ea) — WEP) — DO Wt) (tha — €) 


i=0 


n—-1 


i=0 
follows that 
I(fn) 2 TW(T 


7=0 


-_ » W(ti41) (th, in t?) : 


T 


- [ W (t) dt 


_L? as n — oo. Indeed, by the classical inequality baie == as] <n ey la;|° 


id by the Cauchy-Schwartz inequality 


n—l 
E 


W (tii). (tees _ t?) 


1=0 


-[ wo dt 


} 
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x 
S 
legit 
es) 
en 


7=0 
n—1 2 3 thay s 
i=0 i 
n—-1 3 n—1 
= (t,.-) a 
= iP ee ea oo as 1 — OOo. 


This proves the equality in the exercise. 


Solution 7.10 


Using the same partition of [0,7] into n equal parts as in Solution 7.8 and 
putting 


n—-1 
_ 2 
7=0 


we obtain a sequence fj, fo,... € Mien of random step processes which ap- 
proximates f = 1jo,7,W?. Indeed, 


oo n—l tren ) 
e(/ (0) - fa(OP at) -> A E (\w(t)? — wer)? | ) dt 
i=0 i 
n—1 m 
bi a (3 (t — tf)? + 4(t - ee? ) dt 

1=0 t? 

paatad FA b oe Tak 
: > |) +2(7) | 
= M™ 50 as n — OO. 

nm 


The expectation above is computed with the aid of the following formula valid 
for anyO<s<t: 


E ((W? - w?)”) 


E ((w. 2 W.)*) +4E ((W. ~ W,)° W.) 
4+4E ((W. ~W,)° Ww?) 
3(t—s)°+4(t—s)s 
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‘sing the identity in the hint, we can write 


n—l1 
I(fn) = >> WEP)? (W(t?) — WEP)) 
i=0 
n—-1 
=5 De (WU) - WEY) 
ee ; 1 n—-1 
Pe W(t?) (W(t2,,) — W(t)” 3 (W(t?) 
7=0 1=0 
l : n—1 
= Wy — WEED (ts 2) 
n—-l1 
- weer [(weta) - ween)? = (a - #)] 
i=0 
| lee a 3 
aa 3 (W (t?,,) = W(t; )) 
i=0 


he L? limits of the last three sums are 


n—-1 T 
lim W(t?) (th — tP) 


n—0o 
i=0 0 


Jim >» W(t?) (Ww ( (t7.4) — wt?))* — (th, t?)] = 0 


n—-1 


Jim So (W(t) — WER)? 
i=0 


II 
=) 


ideed, the first limit is correct because 
n—-1 T 
E||S> we?) (ti. -#?) - / W(t) dt 
= 0 
m-1l thy, 
3 of (W(t?) — W(t) at 
i” 


= > E (|w(t?) - wie)l?) dt 
= 34 i (t — t") dt 


T? 
= —>Q asnow. 
2n 


W(t) dt 
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} 
‘) 

| 


= Sewn?) £ (|e) - WY)? - (a - 8) 


1=0 


To the second limit can be verified as follows: 


| 


. > = (wer)? (W(t) S wit?))” “= (tt - t?) 


we) [(WCHR) — WOR))? — (ta — #2) 


: 
n—-1 : 

= 25 tF ( ete 
7=0 


(n — 1) 


; T* 30 asn—-oo. 
n 


Finally, for the third limit we have 


n—-1 
= SE ((W (t2,.1) ~ Wt! ))*) 
i=0 
n—-l1 


lI 

(o>) 
—_ 
= 
+ 
—_ 

~ 
3 
— 

iS) 


1=0 
-1 
2" 65? 
= y= =a 0 as n — © 
97° 1 


It follows that z 
(fa) + gW(T)? - / W (t) dt, 
0) 


which proves the formula in the exercise. 


Solution 7.11 
We shall use part 2) of Theorem 7.1 to verify that 


é(t) = i W(s) dW(s) 


belongs to M? for any T > 0. By Theorem 7.4 €(t) can be identified with an 
adapted modification having a.s. continuous trajectories. It suffices to verify 
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iat €(t) satisfies condition (7.9). Since the stochastic integral is an isometry, 


2 


2 t t 
E =E| Ww (s)Pds = | da: 
0 0 2 


[ W(s) dW(s) 


follows that 


Bf ora=e 


>. €(t) satisfies (7.9). As a result, €(t) belongs to M2. 


ve 


2 T 42 3 
t T 
| 2 gore 


[ W (s)dW(s) 


olution 7.12 


Te shall use the equality proved in Exercise 7.10: 
T T 
W(Ty = 3 | W (t) dt + a} W (t)*dW (t). 
0 0 


he process 3W(t) belongs to Li for any T > 0 because it is adapted and 
aS a.s. continuous paths, so the integral J, |3w()| at exists and is finite. 
y Exercise 7.7 the process 3W(t)? belongs to M? for any T > 0. It follows 
iat W(t)? is an It6 process. Moreover, the above equation can be written in 


ifferential form as 
dW(t)? = 3W(t) dt + 3W(t)? dW (2), 
hich gives a formula. for the stochastic differential dW (t)°. 


olution 7.13 


. has been shown in Exercise 7.9 that 
7 7 
TW(T) = W (t) ar+ | t dW (t). 
0 0 


ince the Wiener process W(t) is adapted and has continuous paths, it belongs 
) £},, while the deterministic process f(t) = t belongs to M4 for any T > 0. 
follows that tW(t) is an It6 process with stochastic differential 


d(tW(t)) = W(t) dt + tdW(t). 


olution 7.14 


or F(t,xz) = e*e~% the partial derivatives are Fi(t,c)= —here-2, Fi(t,z) = 
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ete~% and Fl. (t,z) = e*e~2. Since X(t) = eW(e-2, the Ité formula implies 
that 


dX(t) = dF(t, W(t)) 
(Fete, W(t)) + Ft wi) dt + Fi(t,W(t)) dW(t) 


(~5xt # 5X()) dt + X(t) dW(t) 


X(t) dW (t). 


II 


Because of this, to show that X(t) is an It6 process we need to verify that 
X(t) = e~“e-* belongs to M2 for any T > 0. Clearly, it is an adapted 
process. It was computed in Solution 6.35 that Ee!) = e2, so 


T T ; T 
E | Xia =f Be erkat = | dt = T <0, 
0 0 0 


which proves that X(t) belongs to M2. 


Solution 7.15 


Take F(t,z) = 2". Then Fj(t,z) =0, Fi(t,r) = na"! and F‘(t,z) = n(n - 
1)z"~*. The derivatives of F(t,x) are obviously continuous, so we only need 
to verify that Fi (t,W(t)) = nW(t)"—! belongs to M4 for T > 0. Clearly, it is 
adapted and has a.s. continuous paths. Moreover, 


T T T 
E | jnW (t)?1|° dt = n? | E|w(t)/?"~? dt = i d2n—2t” | dt < ov, 
0 0 0 


where a, = 2*/?q_—1/27( 441) and I'(x) = fo°t?—te7' dt is the Euler gamma 
function. It follows by part 2) of Theorem 7.1 that F’(t,W(t)) = nW(t)""} 
belongs to M#. Therefore we can apply the It6 formula to get 


(n — 1) 


d(w(t)") == W(t)" Pat + nW (t)"-"aW (2), 


as required. 


Solution 7.16 


Some elementary calculus shows that F(t,rz) = e~%'x has continuous partial 
derivatives such that Fy(t,r) = —ae~%z, Fi (t,x) = e * and Fi (t,x) = 0. 
Clearly, €(t) =o i. e~* dW(s) is an It6 process with 


d&(t) = ce“ dW (t). 
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ce the function oe*' F/(t,z) is bounded on each set of the form [0,7] x R, 
ollows immediately that ce’ Fi (t, €(t)) belongs to M2 for any T > 0. Asa 
sequence, we can use the It6 formula (the general case in Theorem 7.6) to 


dY (t) 


d (e~ E(t) 
—ae€(t) dt + e~“ae™ dW (t) 
= -—aY(t)dt+adWi(t), 


ich proves that Y(t) satisfies the equality 
dY (t) = —aY (t) dt+adWi(t). 
lution 7.17 
ce F(t, 2) = e®x and consider the process 
Y(t) = F(t, X(t)). 
2 Y(0) = zo and 
dY(t) = dF(t, X(t)) 
(Fe, X (t)) — aX (t) Fi (t, X (t)) + 507 Feelt, X(d) dt 


2 
+ oF. (t, X(t)) dW(t) 
(ae™ X(t) — ae™ X(t) ) dt + ae dW(t) 
ce™ dW(t). 


the Ito formula. It follows that 


Y(t) = 20 + of e“* dW(s) 
0 


X(t) = e Y(t) 
t 
= e 2 +oenet | e** dW(s). 
0 


queness follows directly from the above argument, but Theorem 7.7 can 
» be used. Namely, the stochastic differential equation 


dX (t) = —aX(t) dt +adWi(t) 


if the form (7.24) with f{z) = —az and g(x) = o, which are Lipschitz 
apie functions. Therefore, the solution to the initial value problem must 


eens meen te than aAlane AF TAA nnannnnana urith aia anntiniuanand noatha 
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Solution 7.18 


According to the theory of ordinary differential equations, (7.31) with initial 
condition x(0) = zp has a unique solution. If z9 = 0, then x(t) = O is the 
solution. If rp # 0, then 

x(t) 


In —— = at + w(t) 
ZO 


by integrating the equation in the hint, which implies that 
z(t) = apes tow), 
Solution 7.19 


By the It6 formula (verify the assumptions!) 


dX(t) = d (Xpe tw) 


2 
(axon a Koei) dt + bX etttoW (h) dW (t) 


2 
(. + 7) X (t) dt + bX (t) dW (Et). 
This proves that X(t) satisfies the stochastic differential equation (7.34). As 


regards the initial condition, we have 


X(0) es XoetttoW = Xo. 
t=0 


Solution 7.20 
The stochastic differential equation 
dX (t) = aX (t) dt + bX(t) dW(t) 


can be written as 


b2 
dxX(t) = (< + 5) X (t) dt + bX (t) dW(E), 


where c = a — a By Exercise 7.19 the solution this stochastic differential 
equation with initial condition X(0) = Xp is 

X(t) = Xpettowt) 
Xpe lan’ )erow lo 


The uniqueness of this solution follows immediately from Theorem 7.7. 
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ution 7.21 
e can write the ordinary differential equation to be solved in the form 


Tea = (1+w'(t)) dt, 
lich implies that 
sinh! x(t) — sinh7' a = t+ w(t). 
ymposing the last formula with sinh, we obtain 
z(t) =sinh(e+t+wi(t)), (7.35) 
vere c = sinh? zo. 
ution 7.22 
nce F(t,z) = sinh(t + x) satisfies the assumptions of the It6 formula, 
dX(t) = dF(t,C+W*(t)) 
= (Fie. + W(t)) + sii (t C+ w()) dt 

+ Fi(t,C + W(t)) dW(t) 
= (cosh(c +t+W(t)) + = sinh(C +t+ w(e)) dt 

+ cosh(C' + t+ W(t)) dW(t) 


(v1 + sinh?(C +t+W(t)) + 5 sinh(C rare wo) dt 


+4/1+sinh*(C +¢+W(t)) dW(t) 
= (vi + X(t)? + 5x1) dt + ( 1 +X(#?] dW (t). 


ne initial condition X(0) = sinh C = Xp is also satisfied. 


a.s. see almost surely 

absolutely continuous distribution 5 

adapted process 140 

adapted sequence of random variables 
48 

almost surely 2 

aperiodic state 105 
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Borel sets 2 
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— and independence 29 

— given a discrete random variable 19 
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